[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260807 07:59:21.566184 11825 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.140.126:39103
I20260807 07:59:21.567646 11825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 07:59:21.568734 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:59:21.598413 11832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:21.598466 11836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:21.599684 11825 server_base.cc:1034] running on GCE node
W20260807 07:59:21.598532 11831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:22.800926 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 07:59:22.801162 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 07:59:22.801407 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089562801385 us; error 0 us; skew 500 ppm
I20260807 07:59:22.802403 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:59:22.810894 11825 webserver.cc:460] Webserver started at http://127.11.140.126:33375/ using document root <none> and password file <none>
I20260807 07:59:22.811766 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 07:59:22.812000 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:59:22.812449 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:59:22.819041 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/master-0-root/instance:
uuid: "7e69eef013a14ace8857d22c2b2461fd"
format_stamp: "Formatted at 2026-08-07 07:59:22 on dist-test-slave-zr1t"
I20260807 07:59:22.826822 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260807 07:59:22.831835 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:22.832854 11825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 07:59:22.833194 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/master-0-root
uuid: "7e69eef013a14ace8857d22c2b2461fd"
format_stamp: "Formatted at 2026-08-07 07:59:22 on dist-test-slave-zr1t"
I20260807 07:59:22.833570 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:59:22.876241 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:59:22.877980 11825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 07:59:22.878515 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:59:22.949405 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.126:39103
I20260807 07:59:22.949506 11928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.126:39103 every 8 connection(s)
I20260807 07:59:22.956881 11929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:22.980648 11929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd: Bootstrap starting.
I20260807 07:59:22.986747 11929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:22.988548 11929 log.cc:824] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd: Log is configured to *not* fsync() on all Append() calls
I20260807 07:59:22.993033 11929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd: No bootstrap required, opened a new log
I20260807 07:59:23.015396 11929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eef013a14ace8857d22c2b2461fd" member_type: VOTER }
I20260807 07:59:23.015929 11929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:23.016237 11929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e69eef013a14ace8857d22c2b2461fd, State: Initialized, Role: FOLLOWER
I20260807 07:59:23.017131 11929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eef013a14ace8857d22c2b2461fd" member_type: VOTER }
I20260807 07:59:23.017663 11929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:59:23.017953 11929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:59:23.018289 11929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:23.022291 11929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eef013a14ace8857d22c2b2461fd" member_type: VOTER }
I20260807 07:59:23.023001 11929 leader_election.cc:302] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e69eef013a14ace8857d22c2b2461fd; no voters:
I20260807 07:59:23.024659 11929 leader_election.cc:288] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:59:23.025014 11934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:23.027343 11934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 1 LEADER]: Becoming Leader. State: Replica: 7e69eef013a14ace8857d22c2b2461fd, State: Running, Role: LEADER
I20260807 07:59:23.028288 11934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eef013a14ace8857d22c2b2461fd" member_type: VOTER }
I20260807 07:59:23.029345 11929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:59:23.040026 11935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e69eef013a14ace8857d22c2b2461fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eef013a14ace8857d22c2b2461fd" member_type: VOTER } }
I20260807 07:59:23.038990 11937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e69eef013a14ace8857d22c2b2461fd. Latest consensus state: current_term: 1 leader_uuid: "7e69eef013a14ace8857d22c2b2461fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eef013a14ace8857d22c2b2461fd" member_type: VOTER } }
I20260807 07:59:23.040729 11935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [sys.catalog]: This master's current role is: LEADER
I20260807 07:59:23.041381 11937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [sys.catalog]: This master's current role is: LEADER
I20260807 07:59:23.049701 11951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:59:23.062029 11951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:59:23.064227 11825 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260807 07:59:23.079362 11951 catalog_manager.cc:1347] Generated new cluster ID: 2634a05e481d473b992a24c0c5516969
I20260807 07:59:23.079649 11951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:59:23.103405 11951 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:59:23.106139 11951 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:59:23.122750 11951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd: Generated new TSK 0
I20260807 07:59:23.124011 11951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:59:23.130656 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:59:23.139529 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:23.140688 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:23.144443 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:23.144621 11825 server_base.cc:1034] running on GCE node
I20260807 07:59:23.145430 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 07:59:23.145714 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 07:59:23.145938 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089563145915 us; error 0 us; skew 500 ppm
I20260807 07:59:23.146669 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:59:23.150406 11825 webserver.cc:460] Webserver started at http://127.11.140.65:36241/ using document root <none> and password file <none>
I20260807 07:59:23.150945 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 07:59:23.151134 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:59:23.151425 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:59:23.152679 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-0-root/instance:
uuid: "e3cb34634f834e639999e07130187758"
format_stamp: "Formatted at 2026-08-07 07:59:23 on dist-test-slave-zr1t"
I20260807 07:59:23.158793 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 07:59:23.163333 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:23.164155 11825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 07:59:23.164431 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-0-root
uuid: "e3cb34634f834e639999e07130187758"
format_stamp: "Formatted at 2026-08-07 07:59:23 on dist-test-slave-zr1t"
I20260807 07:59:23.164700 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:59:23.188342 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:59:23.189646 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:59:23.192466 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:59:23.197178 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:59:23.197384 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:23.197729 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:59:23.197888 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:23.299571 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.65:43471
I20260807 07:59:23.299711 12063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.65:43471 every 8 connection(s)
I20260807 07:59:23.310438 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:59:23.323246 12074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:23.326092 12075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:23.344344 12082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:23.348176 11825 server_base.cc:1034] running on GCE node
I20260807 07:59:23.349146 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260807 07:59:23.349282 12064 heartbeater.cc:347] Connected to a master server at 127.11.140.126:39103
W20260807 07:59:23.349429 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 07:59:23.349669 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089563349652 us; error 0 us; skew 500 ppm
I20260807 07:59:23.349750 12064 heartbeater.cc:464] Registering TS with master...
I20260807 07:59:23.350512 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:59:23.350845 12064 heartbeater.cc:511] Master 127.11.140.126:39103 requested a full tablet report, sending...
I20260807 07:59:23.353693 11825 webserver.cc:460] Webserver started at http://127.11.140.66:45957/ using document root <none> and password file <none>
I20260807 07:59:23.354295 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 07:59:23.354568 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:59:23.354596 11873 ts_manager.cc:194] Registered new tserver with Master: e3cb34634f834e639999e07130187758 (127.11.140.65:43471)
I20260807 07:59:23.354875 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:59:23.356036 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-1-root/instance:
uuid: "52f278a746734ec1a96bce52e7bbba50"
format_stamp: "Formatted at 2026-08-07 07:59:23 on dist-test-slave-zr1t"
I20260807 07:59:23.357719 11873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34014
I20260807 07:59:23.361905 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 07:59:23.366029 12089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:23.366912 11825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 07:59:23.367192 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-1-root
uuid: "52f278a746734ec1a96bce52e7bbba50"
format_stamp: "Formatted at 2026-08-07 07:59:23 on dist-test-slave-zr1t"
I20260807 07:59:23.367466 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:59:23.395355 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:59:23.396548 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:59:23.398677 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:59:23.401432 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:59:23.401708 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:23.401942 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:59:23.402105 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:23.502379 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.66:33089
I20260807 07:59:23.502525 12177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.66:33089 every 8 connection(s)
I20260807 07:59:23.509061 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:59:23.517431 12183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:23.519315 12185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:23.526950 12179 heartbeater.cc:347] Connected to a master server at 127.11.140.126:39103
I20260807 07:59:23.527383 12179 heartbeater.cc:464] Registering TS with master...
I20260807 07:59:23.528352 12179 heartbeater.cc:511] Master 127.11.140.126:39103 requested a full tablet report, sending...
W20260807 07:59:23.528584 12188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:23.530592 11873 ts_manager.cc:194] Registered new tserver with Master: 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:23.532186 11873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34030
I20260807 07:59:23.533814 11825 server_base.cc:1034] running on GCE node
I20260807 07:59:23.534888 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 07:59:23.535146 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 07:59:23.535343 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089563535326 us; error 0 us; skew 500 ppm
I20260807 07:59:23.535921 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:59:23.538743 11825 webserver.cc:460] Webserver started at http://127.11.140.67:44511/ using document root <none> and password file <none>
I20260807 07:59:23.539256 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 07:59:23.539471 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:59:23.539731 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:59:23.540832 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-2-root/instance:
uuid: "31679f1015b349cf941d1a73f5786d47"
format_stamp: "Formatted at 2026-08-07 07:59:23 on dist-test-slave-zr1t"
I20260807 07:59:23.545665 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260807 07:59:23.548938 12193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:23.549762 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:59:23.550040 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-2-root
uuid: "31679f1015b349cf941d1a73f5786d47"
format_stamp: "Formatted at 2026-08-07 07:59:23 on dist-test-slave-zr1t"
I20260807 07:59:23.550323 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786089561396890-11825-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:59:23.569111 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:59:23.570353 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:59:23.571609 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:59:23.574349 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:59:23.574566 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:23.574788 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:59:23.574961 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:23.670068 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.67:33147
I20260807 07:59:23.670163 12281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.67:33147 every 8 connection(s)
I20260807 07:59:23.688220 12282 heartbeater.cc:347] Connected to a master server at 127.11.140.126:39103
I20260807 07:59:23.688609 12282 heartbeater.cc:464] Registering TS with master...
I20260807 07:59:23.689307 12282 heartbeater.cc:511] Master 127.11.140.126:39103 requested a full tablet report, sending...
I20260807 07:59:23.691185 11873 ts_manager.cc:194] Registered new tserver with Master: 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:23.692071 11825 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017238639s
I20260807 07:59:23.692711 11873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34040
I20260807 07:59:23.695930 11825 test_util.cc:268] Using random seed: 1643850527
I20260807 07:59:23.750100 11873 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34042:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:59:23.754447 11873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:59:23.879279 12237 tablet_service.cc:1467] Processing CreateTablet for tablet f442cf60f9a64b8398f1fa3db587c360 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260807 07:59:23.881865 12233 tablet_service.cc:1467] Processing CreateTablet for tablet 289aa3803f5a417e81efe7b311171d6d (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260807 07:59:23.885380 12015 tablet_service.cc:1467] Processing CreateTablet for tablet f442cf60f9a64b8398f1fa3db587c360 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260807 07:59:23.889389 12237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f442cf60f9a64b8398f1fa3db587c360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.889297 12014 tablet_service.cc:1467] Processing CreateTablet for tablet f5bc59ebb68049018d4a79e610014811 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260807 07:59:23.892944 12231 tablet_service.cc:1467] Processing CreateTablet for tablet c972c80c939942feb0b61190b3d16449 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260807 07:59:23.891439 12228 tablet_service.cc:1467] Processing CreateTablet for tablet b205118d55de47b185bf281b63fd2841 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260807 07:59:23.887611 12230 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfb176244ad4dddae71d095b33c7326 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260807 07:59:23.898552 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f442cf60f9a64b8398f1fa3db587c360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.894821 12234 tablet_service.cc:1467] Processing CreateTablet for tablet f5bc59ebb68049018d4a79e610014811 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260807 07:59:23.902171 12233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289aa3803f5a417e81efe7b311171d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.901933 12011 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee27aa48f7045a9b753c390eb26aeb4 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260807 07:59:23.910411 12014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bc59ebb68049018d4a79e610014811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.902186 12128 tablet_service.cc:1467] Processing CreateTablet for tablet f442cf60f9a64b8398f1fa3db587c360 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260807 07:59:23.914485 12010 tablet_service.cc:1467] Processing CreateTablet for tablet c972c80c939942feb0b61190b3d16449 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260807 07:59:23.905562 12232 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee27aa48f7045a9b753c390eb26aeb4 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260807 07:59:23.917117 12231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c972c80c939942feb0b61190b3d16449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.923939 12123 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfb176244ad4dddae71d095b33c7326 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260807 07:59:23.927845 12011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee27aa48f7045a9b753c390eb26aeb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.924678 12125 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee27aa48f7045a9b753c390eb26aeb4 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260807 07:59:23.928185 12126 tablet_service.cc:1467] Processing CreateTablet for tablet 289aa3803f5a417e81efe7b311171d6d (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260807 07:59:23.920397 12226 tablet_service.cc:1467] Processing CreateTablet for tablet 28366b85f02b466597fd0583d79b21fb (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260807 07:59:23.937848 12224 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9d77bf75b643218014f3d531b1cd29 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260807 07:59:23.937935 12005 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9d77bf75b643218014f3d531b1cd29 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260807 07:59:23.936491 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f442cf60f9a64b8398f1fa3db587c360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.929702 12008 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfb176244ad4dddae71d095b33c7326 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260807 07:59:23.940629 12120 tablet_service.cc:1467] Processing CreateTablet for tablet 28366b85f02b466597fd0583d79b21fb (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260807 07:59:23.927135 12013 tablet_service.cc:1467] Processing CreateTablet for tablet 289aa3803f5a417e81efe7b311171d6d (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260807 07:59:23.929229 12007 tablet_service.cc:1467] Processing CreateTablet for tablet b205118d55de47b185bf281b63fd2841 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260807 07:59:23.940783 12119 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9d77bf75b643218014f3d531b1cd29 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260807 07:59:23.942910 12124 tablet_service.cc:1467] Processing CreateTablet for tablet c972c80c939942feb0b61190b3d16449 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260807 07:59:23.936806 12127 tablet_service.cc:1467] Processing CreateTablet for tablet f5bc59ebb68049018d4a79e610014811 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260807 07:59:23.928929 12006 tablet_service.cc:1467] Processing CreateTablet for tablet 28366b85f02b466597fd0583d79b21fb (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260807 07:59:23.933476 12122 tablet_service.cc:1467] Processing CreateTablet for tablet b205118d55de47b185bf281b63fd2841 (DEFAULT_TABLE table=test-workload [id=4585abfc988f4bd89c2a13e408033666]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260807 07:59:23.953177 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee27aa48f7045a9b753c390eb26aeb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.954195 12010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c972c80c939942feb0b61190b3d16449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.955431 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289aa3803f5a417e81efe7b311171d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.957296 12007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b205118d55de47b185bf281b63fd2841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.958592 12013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289aa3803f5a417e81efe7b311171d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.959743 12122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b205118d55de47b185bf281b63fd2841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.962025 12119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9d77bf75b643218014f3d531b1cd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.966969 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c972c80c939942feb0b61190b3d16449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.967272 12228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b205118d55de47b185bf281b63fd2841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.968075 12005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9d77bf75b643218014f3d531b1cd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.971737 12232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee27aa48f7045a9b753c390eb26aeb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.972759 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bc59ebb68049018d4a79e610014811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.981122 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bc59ebb68049018d4a79e610014811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.982033 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28366b85f02b466597fd0583d79b21fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.980067 12230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfb176244ad4dddae71d095b33c7326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.984465 12224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9d77bf75b643218014f3d531b1cd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.986743 12226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28366b85f02b466597fd0583d79b21fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.989600 12006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28366b85f02b466597fd0583d79b21fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.992245 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfb176244ad4dddae71d095b33c7326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:23.992359 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfb176244ad4dddae71d095b33c7326. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 07:59:24.055361 11950 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 07:59:24.094135 12304 tablet_bootstrap.cc:492] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47: Bootstrap starting.
I20260807 07:59:24.145164 12305 tablet_bootstrap.cc:492] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50: Bootstrap starting.
I20260807 07:59:24.196646 12306 tablet_bootstrap.cc:492] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758: Bootstrap starting.
I20260807 07:59:24.208660 12304 tablet_bootstrap.cc:654] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.214541 12305 tablet_bootstrap.cc:654] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.218019 12306 tablet_bootstrap.cc:654] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.224184 12305 tablet_bootstrap.cc:492] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50: No bootstrap required, opened a new log
I20260807 07:59:24.224752 12305 ts_tablet_manager.cc:1397] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50: Time spent bootstrapping tablet: real 0.087s user 0.012s sys 0.003s
I20260807 07:59:24.225214 12304 tablet_bootstrap.cc:492] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47: No bootstrap required, opened a new log
I20260807 07:59:24.225652 12304 ts_tablet_manager.cc:1397] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47: Time spent bootstrapping tablet: real 0.149s user 0.009s sys 0.012s
I20260807 07:59:24.228065 12305 raft_consensus.cc:348] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:24.228600 12304 raft_consensus.cc:348] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.229377 12304 raft_consensus.cc:374] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.229873 12304 raft_consensus.cc:729] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.230564 12304 consensus_queue.cc:260] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.231621 12306 tablet_bootstrap.cc:492] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758: No bootstrap required, opened a new log
I20260807 07:59:24.232082 12306 ts_tablet_manager.cc:1397] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.008s
I20260807 07:59:24.233011 12282 heartbeater.cc:503] Master 127.11.140.126:39103 was elected leader, sending a full tablet report...
I20260807 07:59:24.233333 12304 ts_tablet_manager.cc:1428] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 07:59:24.234256 12304 tablet_bootstrap.cc:492] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47: Bootstrap starting.
I20260807 07:59:24.234750 12306 raft_consensus.cc:348] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:24.235424 12306 raft_consensus.cc:374] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.235699 12306 raft_consensus.cc:729] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.236346 12306 consensus_queue.cc:260] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:24.229398 12305 raft_consensus.cc:374] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.238634 12305 raft_consensus.cc:729] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.239293 12305 consensus_queue.cc:260] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:24.240504 12304 tablet_bootstrap.cc:654] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.248375 12179 heartbeater.cc:503] Master 127.11.140.126:39103 was elected leader, sending a full tablet report...
I20260807 07:59:24.253989 12064 heartbeater.cc:503] Master 127.11.140.126:39103 was elected leader, sending a full tablet report...
I20260807 07:59:24.255048 12305 ts_tablet_manager.cc:1428] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50: Time spent starting tablet: real 0.030s user 0.004s sys 0.006s
I20260807 07:59:24.255983 12305 tablet_bootstrap.cc:492] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50: Bootstrap starting.
W20260807 07:59:24.261139 12180 tablet.cc:2369] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:59:24.263415 12305 tablet_bootstrap.cc:654] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.264936 12304 tablet_bootstrap.cc:492] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47: No bootstrap required, opened a new log
I20260807 07:59:24.265476 12304 ts_tablet_manager.cc:1397] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.014s
I20260807 07:59:24.266733 12306 ts_tablet_manager.cc:1428] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758: Time spent starting tablet: real 0.034s user 0.007s sys 0.002s
I20260807 07:59:24.268836 12306 tablet_bootstrap.cc:492] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758: Bootstrap starting.
I20260807 07:59:24.270737 12304 raft_consensus.cc:348] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.271534 12304 raft_consensus.cc:374] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.271867 12304 raft_consensus.cc:729] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.272606 12304 consensus_queue.cc:260] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.274778 12304 ts_tablet_manager.cc:1428] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260807 07:59:24.275593 12306 tablet_bootstrap.cc:654] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.275720 12304 tablet_bootstrap.cc:492] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47: Bootstrap starting.
I20260807 07:59:24.282482 12304 tablet_bootstrap.cc:654] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.284596 12306 tablet_bootstrap.cc:492] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758: No bootstrap required, opened a new log
I20260807 07:59:24.286384 12306 ts_tablet_manager.cc:1397] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20260807 07:59:24.294975 12306 raft_consensus.cc:348] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.295783 12306 raft_consensus.cc:374] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.296114 12306 raft_consensus.cc:729] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.296880 12306 consensus_queue.cc:260] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.300179 12305 tablet_bootstrap.cc:492] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50: No bootstrap required, opened a new log
I20260807 07:59:24.300628 12305 ts_tablet_manager.cc:1397] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.000s
I20260807 07:59:24.301241 12304 tablet_bootstrap.cc:492] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47: No bootstrap required, opened a new log
I20260807 07:59:24.301765 12304 ts_tablet_manager.cc:1397] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.007s
I20260807 07:59:24.302039 12306 ts_tablet_manager.cc:1428] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20260807 07:59:24.303004 12306 tablet_bootstrap.cc:492] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758: Bootstrap starting.
I20260807 07:59:24.303249 12305 raft_consensus.cc:348] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.303995 12305 raft_consensus.cc:374] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.304278 12304 raft_consensus.cc:348] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.304991 12304 raft_consensus.cc:374] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.305325 12304 raft_consensus.cc:729] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.306053 12304 consensus_queue.cc:260] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.308506 12304 ts_tablet_manager.cc:1428] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 07:59:24.309379 12304 tablet_bootstrap.cc:492] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47: Bootstrap starting.
I20260807 07:59:24.332719 12311 raft_consensus.cc:482] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.333381 12311 raft_consensus.cc:504] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.336861 12306 tablet_bootstrap.cc:654] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.304286 12305 raft_consensus.cc:729] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.341354 12305 consensus_queue.cc:260] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.341372 12311 leader_election.cc:288] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:24.345917 12313 raft_consensus.cc:482] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.346370 12313 raft_consensus.cc:504] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.347458 12305 ts_tablet_manager.cc:1428] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50: Time spent starting tablet: real 0.047s user 0.000s sys 0.007s
I20260807 07:59:24.348476 12305 tablet_bootstrap.cc:492] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50: Bootstrap starting.
I20260807 07:59:24.348608 12313 leader_election.cc:288] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.351198 12304 tablet_bootstrap.cc:654] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.359550 12305 tablet_bootstrap.cc:654] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.381336 12306 tablet_bootstrap.cc:492] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758: No bootstrap required, opened a new log
I20260807 07:59:24.381976 12306 ts_tablet_manager.cc:1397] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.006s
I20260807 07:59:24.391119 12304 tablet_bootstrap.cc:492] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47: No bootstrap required, opened a new log
I20260807 07:59:24.385376 12306 raft_consensus.cc:348] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.391472 12306 raft_consensus.cc:374] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.391597 12304 ts_tablet_manager.cc:1397] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47: Time spent bootstrapping tablet: real 0.082s user 0.024s sys 0.036s
I20260807 07:59:24.391781 12306 raft_consensus.cc:729] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.392473 12306 consensus_queue.cc:260] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.395141 12314 raft_consensus.cc:482] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.393638 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.398598 12027 raft_consensus.cc:2454] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 0.
I20260807 07:59:24.395846 12304 raft_consensus.cc:348] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.399295 12304 raft_consensus.cc:374] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.399583 12314 raft_consensus.cc:504] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.401829 12196 leader_election.cc:302] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:24.408568 12305 tablet_bootstrap.cc:492] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50: No bootstrap required, opened a new log
I20260807 07:59:24.409134 12305 ts_tablet_manager.cc:1397] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50: Time spent bootstrapping tablet: real 0.061s user 0.015s sys 0.002s
I20260807 07:59:24.399595 12304 raft_consensus.cc:729] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.410375 12311 raft_consensus.cc:2781] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:24.404121 12314 leader_election.cc:288] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.413831 12311 raft_consensus.cc:482] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:24.414237 12311 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.423262 12311 raft_consensus.cc:504] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.425966 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:24.426831 12027 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.429239 12305 raft_consensus.cc:348] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.430442 12305 raft_consensus.cc:374] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.431613 12305 raft_consensus.cc:729] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.433378 12305 consensus_queue.cc:260] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.436085 12305 ts_tablet_manager.cc:1428] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20260807 07:59:24.436663 12306 ts_tablet_manager.cc:1428] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758: Time spent starting tablet: real 0.054s user 0.007s sys 0.001s
I20260807 07:59:24.427809 12311 leader_election.cc:288] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:24.438321 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50" is_pre_election: true
I20260807 07:59:24.440267 12306 tablet_bootstrap.cc:492] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758: Bootstrap starting.
I20260807 07:59:24.442828 12305 tablet_bootstrap.cc:492] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50: Bootstrap starting.
I20260807 07:59:24.428079 12304 consensus_queue.cc:260] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.450881 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:24.452031 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50" is_pre_election: true
W20260807 07:59:24.455435 11984 leader_election.cc:341] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089): Illegal state: must be running to vote when last-logged opid is not known
W20260807 07:59:24.452663 12197 leader_election.cc:341] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.455914 12304 ts_tablet_manager.cc:1428] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47: Time spent starting tablet: real 0.064s user 0.003s sys 0.007s
W20260807 07:59:24.456880 12197 leader_election.cc:341] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.457204 12304 tablet_bootstrap.cc:492] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47: Bootstrap starting.
I20260807 07:59:24.468623 12305 tablet_bootstrap.cc:654] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.469933 12027 raft_consensus.cc:2454] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 1.
I20260807 07:59:24.473778 12306 tablet_bootstrap.cc:654] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.474586 12304 tablet_bootstrap.cc:654] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.477954 12196 leader_election.cc:302] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters: 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:24.478773 12311 raft_consensus.cc:2781] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:24.484432 12311 raft_consensus.cc:686] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Becoming Leader. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:24.485630 12305 tablet_bootstrap.cc:492] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50: No bootstrap required, opened a new log
I20260807 07:59:24.485287 12311 consensus_queue.cc:237] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.486071 12305 ts_tablet_manager.cc:1397] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.006s
I20260807 07:59:24.488639 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47" is_pre_election: true
I20260807 07:59:24.488699 12305 raft_consensus.cc:348] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.489445 12305 raft_consensus.cc:374] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.489470 12250 raft_consensus.cc:2454] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 0.
I20260807 07:59:24.489786 12305 raft_consensus.cc:729] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.490533 12305 consensus_queue.cc:260] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.493569 11984 leader_election.cc:302] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters: 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:24.494971 12313 raft_consensus.cc:2781] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:24.495325 12313 raft_consensus.cc:482] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:24.496876 12313 raft_consensus.cc:3037] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.494834 12305 ts_tablet_manager.cc:1428] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 07:59:24.505824 12314 raft_consensus.cc:482] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.506362 12314 raft_consensus.cc:504] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.517264 12305 tablet_bootstrap.cc:492] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50: Bootstrap starting.
I20260807 07:59:24.520888 12314 leader_election.cc:288] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.521392 12314 raft_consensus.cc:482] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.521852 12314 raft_consensus.cc:504] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:24.538151 12314 leader_election.cc:288] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147), e3cb34634f834e639999e07130187758 (127.11.140.65:43471)
I20260807 07:59:24.539752 12306 tablet_bootstrap.cc:492] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758: No bootstrap required, opened a new log
I20260807 07:59:24.540843 12306 ts_tablet_manager.cc:1397] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758: Time spent bootstrapping tablet: real 0.101s user 0.018s sys 0.007s
I20260807 07:59:24.542137 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47" is_pre_election: true
I20260807 07:59:24.543514 12305 tablet_bootstrap.cc:654] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.540338 12313 raft_consensus.cc:504] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.544123 12306 raft_consensus.cc:348] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.544822 12306 raft_consensus.cc:374] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.545125 12306 raft_consensus.cc:729] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.545800 12313 leader_election.cc:288] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 1 election: Requested vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.548437 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:24.550591 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47" is_pre_election: true
I20260807 07:59:24.551635 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47" is_pre_election: true
I20260807 07:59:24.545807 12306 consensus_queue.cc:260] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
W20260807 07:59:24.553392 11984 leader_election.cc:341] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089): Illegal state: must be running to vote when last-logged opid is not known
W20260807 07:59:24.552758 12097 leader_election.cc:341] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.554242 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:24.554777 12249 raft_consensus.cc:3037] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.556842 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.555944 12314 raft_consensus.cc:482] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.558825 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.560173 12026 raft_consensus.cc:2454] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 0.
I20260807 07:59:24.558902 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.561378 12306 ts_tablet_manager.cc:1428] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758: Time spent starting tablet: real 0.020s user 0.001s sys 0.006s
I20260807 07:59:24.561609 12025 raft_consensus.cc:2454] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 0.
W20260807 07:59:24.558539 12097 leader_election.cc:341] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.548300 11870 catalog_manager.cc:5495] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31679f1015b349cf941d1a73f5786d47 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "31679f1015b349cf941d1a73f5786d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:24.560586 12314 raft_consensus.cc:504] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.565980 12304 tablet_bootstrap.cc:492] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47: No bootstrap required, opened a new log
W20260807 07:59:24.563974 12097 leader_election.cc:341] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.568571 12304 ts_tablet_manager.cc:1397] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47: Time spent bootstrapping tablet: real 0.112s user 0.017s sys 0.010s
I20260807 07:59:24.571291 12249 raft_consensus.cc:2454] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 1.
I20260807 07:59:24.571899 12304 raft_consensus.cc:348] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.572362 11984 leader_election.cc:302] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters: 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:24.572921 12304 raft_consensus.cc:374] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.573359 12313 raft_consensus.cc:2781] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader election won for term 1
W20260807 07:59:24.573747 12096 leader_election.cc:341] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cb34634f834e639999e07130187758 (127.11.140.65:43471): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.574067 12304 raft_consensus.cc:729] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.577843 12313 raft_consensus.cc:686] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 1 LEADER]: Becoming Leader. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:24.579852 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47" is_pre_election: true
I20260807 07:59:24.577683 12304 consensus_queue.cc:260] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.578392 12314 leader_election.cc:288] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.580224 12305 tablet_bootstrap.cc:492] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50: No bootstrap required, opened a new log
I20260807 07:59:24.580663 12249 raft_consensus.cc:2454] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 0.
I20260807 07:59:24.581003 12305 ts_tablet_manager.cc:1397] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50: Time spent bootstrapping tablet: real 0.065s user 0.012s sys 0.004s
I20260807 07:59:24.580981 12313 consensus_queue.cc:237] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.582655 12304 ts_tablet_manager.cc:1428] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47: Time spent starting tablet: real 0.014s user 0.005s sys 0.003s
I20260807 07:59:24.582710 12097 leader_election.cc:302] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters:
I20260807 07:59:24.581241 12306 tablet_bootstrap.cc:492] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758: Bootstrap starting.
I20260807 07:59:24.583999 12304 tablet_bootstrap.cc:492] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47: Bootstrap starting.
I20260807 07:59:24.585215 12314 raft_consensus.cc:2781] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:24.589074 12314 raft_consensus.cc:482] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:24.588414 12305 raft_consensus.cc:348] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.589644 12305 raft_consensus.cc:374] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.589911 12305 raft_consensus.cc:729] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.576540 12096 leader_election.cc:302] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50; no voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758
I20260807 07:59:24.592856 12344 raft_consensus.cc:2726] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:24.589403 12314 raft_consensus.cc:3037] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.590559 12305 consensus_queue.cc:260] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.596658 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.596526 12096 leader_election.cc:302] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters: 31679f1015b349cf941d1a73f5786d47
I20260807 07:59:24.598559 12344 raft_consensus.cc:2781] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:24.598891 12344 raft_consensus.cc:482] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:24.599206 12344 raft_consensus.cc:3037] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.602299 12096 leader_election.cc:302] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters: 31679f1015b349cf941d1a73f5786d47
I20260807 07:59:24.607226 12305 ts_tablet_manager.cc:1428] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50: Time spent starting tablet: real 0.025s user 0.002s sys 0.006s
I20260807 07:59:24.608129 12305 tablet_bootstrap.cc:492] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50: Bootstrap starting.
I20260807 07:59:24.605077 11865 catalog_manager.cc:5495] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cb34634f834e639999e07130187758 (127.11.140.65). New cstate: current_term: 1 leader_uuid: "e3cb34634f834e639999e07130187758" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:24.611174 12306 tablet_bootstrap.cc:654] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.619673 12350 raft_consensus.cc:2781] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:24.620327 12350 raft_consensus.cc:482] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:24.620687 12350 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Advancing to term 1
W20260807 07:59:24.621018 12096 leader_election.cc:341] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cb34634f834e639999e07130187758 (127.11.140.65:43471): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.625725 12305 tablet_bootstrap.cc:654] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.632740 12304 tablet_bootstrap.cc:654] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.640111 12344 raft_consensus.cc:504] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.646595 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:24.650058 12344 leader_election.cc:288] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.652545 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:24.648041 12025 raft_consensus.cc:3037] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Advancing to term 1
W20260807 07:59:24.656687 12097 leader_election.cc:341] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.659502 12304 tablet_bootstrap.cc:492] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47: No bootstrap required, opened a new log
I20260807 07:59:24.659087 12350 raft_consensus.cc:504] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:24.660204 12304 ts_tablet_manager.cc:1397] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.005s
I20260807 07:59:24.660995 12314 raft_consensus.cc:504] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.663981 12305 tablet_bootstrap.cc:492] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50: No bootstrap required, opened a new log
I20260807 07:59:24.664422 12305 ts_tablet_manager.cc:1397] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.011s
I20260807 07:59:24.664709 12025 raft_consensus.cc:2454] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 1.
I20260807 07:59:24.664948 12304 raft_consensus.cc:348] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:24.665838 12096 leader_election.cc:302] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters: 31679f1015b349cf941d1a73f5786d47
I20260807 07:59:24.666186 12304 raft_consensus.cc:374] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.667183 12344 raft_consensus.cc:2781] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:24.666631 12304 raft_consensus.cc:729] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.670522 12350 leader_election.cc:288] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Requested vote from peers 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147), e3cb34634f834e639999e07130187758 (127.11.140.65:43471)
I20260807 07:59:24.671171 12306 tablet_bootstrap.cc:492] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758: No bootstrap required, opened a new log
I20260807 07:59:24.672009 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:24.672595 12306 ts_tablet_manager.cc:1397] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758: Time spent bootstrapping tablet: real 0.106s user 0.013s sys 0.013s
I20260807 07:59:24.672339 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:24.669409 12304 consensus_queue.cc:260] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:24.673627 12026 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.674664 12314 leader_election.cc:288] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.669342 12305 raft_consensus.cc:348] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.677492 12305 raft_consensus.cc:374] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.678032 12305 raft_consensus.cc:729] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.674084 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:24.679159 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:24.679714 12250 raft_consensus.cc:3037] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.680104 12304 ts_tablet_manager.cc:1428] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260807 07:59:24.675348 12249 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Advancing to term 1
W20260807 07:59:24.691730 12096 leader_election.cc:341] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3cb34634f834e639999e07130187758 (127.11.140.65:43471): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.691797 12305 consensus_queue.cc:260] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.695498 12306 raft_consensus.cc:348] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.696894 12306 raft_consensus.cc:374] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.695509 12304 tablet_bootstrap.cc:492] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47: Bootstrap starting.
I20260807 07:59:24.697284 12306 raft_consensus.cc:729] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.700783 12026 raft_consensus.cc:2454] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 1.
I20260807 07:59:24.701289 12306 consensus_queue.cc:260] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.702380 12096 leader_election.cc:302] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:24.708473 12250 raft_consensus.cc:2454] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 1.
I20260807 07:59:24.708673 12314 raft_consensus.cc:2781] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:24.709481 12314 raft_consensus.cc:686] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:24.710362 12097 leader_election.cc:302] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters: e3cb34634f834e639999e07130187758
I20260807 07:59:24.711753 12350 raft_consensus.cc:2781] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:24.712196 12350 raft_consensus.cc:686] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:24.712222 12249 raft_consensus.cc:2454] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 1.
I20260807 07:59:24.710557 12314 consensus_queue.cc:237] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:24.716564 12350 consensus_queue.cc:237] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.722551 12344 raft_consensus.cc:686] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:24.726632 12305 ts_tablet_manager.cc:1428] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50: Time spent starting tablet: real 0.062s user 0.007s sys 0.003s
I20260807 07:59:24.728526 12306 ts_tablet_manager.cc:1428] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758: Time spent starting tablet: real 0.056s user 0.011s sys 0.000s
I20260807 07:59:24.730489 12344 consensus_queue.cc:237] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.734862 12306 tablet_bootstrap.cc:492] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758: Bootstrap starting.
I20260807 07:59:24.735881 12305 tablet_bootstrap.cc:492] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50: Bootstrap starting.
I20260807 07:59:24.766640 12304 tablet_bootstrap.cc:654] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.767596 12305 tablet_bootstrap.cc:654] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.772013 12304 tablet_bootstrap.cc:492] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47: No bootstrap required, opened a new log
I20260807 07:59:24.772495 12304 ts_tablet_manager.cc:1397] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47: Time spent bootstrapping tablet: real 0.077s user 0.013s sys 0.003s
I20260807 07:59:24.775290 12306 tablet_bootstrap.cc:654] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.775550 12304 raft_consensus.cc:348] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.776203 12305 tablet_bootstrap.cc:492] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50: No bootstrap required, opened a new log
I20260807 07:59:24.776338 12304 raft_consensus.cc:374] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.776661 12305 ts_tablet_manager.cc:1397] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.001s
I20260807 07:59:24.776639 12304 raft_consensus.cc:729] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.777482 12304 consensus_queue.cc:260] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.780750 12304 ts_tablet_manager.cc:1428] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47: Time spent starting tablet: real 0.008s user 0.002s sys 0.007s
I20260807 07:59:24.780269 12305 raft_consensus.cc:348] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.781226 12305 raft_consensus.cc:374] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.781662 12305 raft_consensus.cc:729] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.781941 12304 tablet_bootstrap.cc:492] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47: Bootstrap starting.
I20260807 07:59:24.782032 12306 tablet_bootstrap.cc:492] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758: No bootstrap required, opened a new log
I20260807 07:59:24.782742 12306 ts_tablet_manager.cc:1397] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.011s
I20260807 07:59:24.782580 12305 consensus_queue.cc:260] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.785185 12305 ts_tablet_manager.cc:1428] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260807 07:59:24.786156 12305 tablet_bootstrap.cc:492] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50: Bootstrap starting.
I20260807 07:59:24.786140 12306 raft_consensus.cc:348] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.787117 12306 raft_consensus.cc:374] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.787453 12306 raft_consensus.cc:729] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.788151 12306 consensus_queue.cc:260] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.792456 12305 tablet_bootstrap.cc:654] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.792290 11865 catalog_manager.cc:5495] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 1 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:24.807654 12304 tablet_bootstrap.cc:654] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.822212 12304 tablet_bootstrap.cc:492] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47: No bootstrap required, opened a new log
I20260807 07:59:24.822701 12304 ts_tablet_manager.cc:1397] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47: Time spent bootstrapping tablet: real 0.041s user 0.016s sys 0.000s
I20260807 07:59:24.823378 12311 raft_consensus.cc:482] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.823872 12311 raft_consensus.cc:504] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.827198 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.825843 12311 leader_election.cc:288] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:24.827767 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50" is_pre_election: true
W20260807 07:59:24.829484 12196 leader_election.cc:341] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cb34634f834e639999e07130187758 (127.11.140.65:43471): Illegal state: must be running to vote when last-logged opid is not known
W20260807 07:59:24.830269 12197 leader_election.cc:341] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.830807 12197 leader_election.cc:302] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47; no voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758
I20260807 07:59:24.831754 12311 raft_consensus.cc:2726] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:24.827909 11869 catalog_manager.cc:5495] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 1 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:24.832638 12304 raft_consensus.cc:348] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.833334 12304 raft_consensus.cc:374] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.833674 12304 raft_consensus.cc:729] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.834345 12304 consensus_queue.cc:260] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.825277 12306 ts_tablet_manager.cc:1428] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758: Time spent starting tablet: real 0.042s user 0.006s sys 0.001s
I20260807 07:59:24.836650 12304 ts_tablet_manager.cc:1428] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260807 07:59:24.837734 12304 tablet_bootstrap.cc:492] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47: Bootstrap starting.
I20260807 07:59:24.842715 12304 tablet_bootstrap.cc:654] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.849009 12306 tablet_bootstrap.cc:492] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758: Bootstrap starting.
I20260807 07:59:24.853650 12304 tablet_bootstrap.cc:492] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47: No bootstrap required, opened a new log
I20260807 07:59:24.854046 12304 ts_tablet_manager.cc:1397] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260807 07:59:24.856182 12304 raft_consensus.cc:348] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.856853 12304 raft_consensus.cc:374] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.857098 12304 raft_consensus.cc:729] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.857869 12304 consensus_queue.cc:260] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.859982 12304 ts_tablet_manager.cc:1428] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260807 07:59:24.834112 11869 catalog_manager.cc:5495] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 1 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:24.873641 12306 tablet_bootstrap.cc:654] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.876905 12305 tablet_bootstrap.cc:492] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50: No bootstrap required, opened a new log
I20260807 07:59:24.877358 12305 ts_tablet_manager.cc:1397] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50: Time spent bootstrapping tablet: real 0.091s user 0.034s sys 0.049s
I20260807 07:59:24.879910 12305 raft_consensus.cc:348] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.880623 12305 raft_consensus.cc:374] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.880911 12305 raft_consensus.cc:729] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.881541 12305 consensus_queue.cc:260] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.883518 12305 ts_tablet_manager.cc:1428] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 07:59:24.884290 12305 tablet_bootstrap.cc:492] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50: Bootstrap starting.
I20260807 07:59:24.889464 12305 tablet_bootstrap.cc:654] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.890936 12344 raft_consensus.cc:482] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.891387 12344 raft_consensus.cc:504] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.892555 12306 tablet_bootstrap.cc:492] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758: No bootstrap required, opened a new log
I20260807 07:59:24.893359 12344 leader_election.cc:288] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.894073 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c972c80c939942feb0b61190b3d16449" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.894394 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c972c80c939942feb0b61190b3d16449" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47" is_pre_election: true
I20260807 07:59:24.895040 12249 raft_consensus.cc:2454] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 0.
W20260807 07:59:24.895859 12096 leader_election.cc:341] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cb34634f834e639999e07130187758 (127.11.140.65:43471): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.896888 12097 leader_election.cc:302] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters: e3cb34634f834e639999e07130187758
I20260807 07:59:24.898000 12344 raft_consensus.cc:2781] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:24.898375 12344 raft_consensus.cc:482] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:24.898707 12344 raft_consensus.cc:3037] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.900313 12305 tablet_bootstrap.cc:492] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50: No bootstrap required, opened a new log
I20260807 07:59:24.900677 12305 ts_tablet_manager.cc:1397] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260807 07:59:24.902595 12305 raft_consensus.cc:348] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.903079 12305 raft_consensus.cc:374] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.903288 12305 raft_consensus.cc:729] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.903812 12305 consensus_queue.cc:260] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.905238 12305 ts_tablet_manager.cc:1428] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 07:59:24.904920 12344 raft_consensus.cc:504] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.907778 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c972c80c939942feb0b61190b3d16449" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:24.907980 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c972c80c939942feb0b61190b3d16449" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:24.908550 12249 raft_consensus.cc:3037] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Advancing to term 1
W20260807 07:59:24.908819 12096 leader_election.cc:341] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3cb34634f834e639999e07130187758 (127.11.140.65:43471): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:24.913621 12306 ts_tablet_manager.cc:1397] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.005s
I20260807 07:59:24.914834 12249 raft_consensus.cc:2454] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 1.
I20260807 07:59:24.907374 12344 leader_election.cc:288] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.915897 12097 leader_election.cc:302] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters: e3cb34634f834e639999e07130187758
I20260807 07:59:24.916188 12306 raft_consensus.cc:348] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.916661 12344 raft_consensus.cc:2781] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:24.916914 12306 raft_consensus.cc:374] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.917204 12306 raft_consensus.cc:729] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.917176 12344 raft_consensus.cc:686] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:24.917923 12306 consensus_queue.cc:260] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.920197 12306 ts_tablet_manager.cc:1428] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 07:59:24.918579 12344 consensus_queue.cc:237] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.921061 12306 tablet_bootstrap.cc:492] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758: Bootstrap starting.
I20260807 07:59:24.924790 12311 raft_consensus.cc:482] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.925796 12313 raft_consensus.cc:482] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.926204 12313 raft_consensus.cc:504] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.928010 12313 leader_election.cc:288] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.928341 12306 tablet_bootstrap.cc:654] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.928550 12339 raft_consensus.cc:482] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.928956 12339 raft_consensus.cc:504] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.930123 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50" is_pre_election: true
I20260807 07:59:24.925217 12311 raft_consensus.cc:504] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.933545 12311 leader_election.cc:288] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:24.933925 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50" is_pre_election: true
I20260807 07:59:24.933117 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47" is_pre_election: true
I20260807 07:59:24.934705 12339 leader_election.cc:288] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:24.935883 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50" is_pre_election: true
I20260807 07:59:24.937288 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.937793 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.938222 12026 raft_consensus.cc:2379] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31679f1015b349cf941d1a73f5786d47 in current term 1: Already voted for candidate 52f278a746734ec1a96bce52e7bbba50 in this term.
I20260807 07:59:24.938406 12025 raft_consensus.cc:2454] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 0.
I20260807 07:59:24.937346 12140 raft_consensus.cc:2454] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 0.
I20260807 07:59:24.939996 12196 leader_election.cc:302] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47; no voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758
I20260807 07:59:24.940919 12249 raft_consensus.cc:2379] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3cb34634f834e639999e07130187758 in current term 1: Already voted for candidate 52f278a746734ec1a96bce52e7bbba50 in this term.
I20260807 07:59:24.940860 12197 leader_election.cc:302] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters:
I20260807 07:59:24.942145 11984 leader_election.cc:302] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cb34634f834e639999e07130187758; no voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:24.943207 12313 raft_consensus.cc:3037] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.942456 12311 raft_consensus.cc:3037] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.945749 11865 catalog_manager.cc:5495] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 1 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:24.947853 12306 tablet_bootstrap.cc:492] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758: No bootstrap required, opened a new log
I20260807 07:59:24.947038 12339 raft_consensus.cc:2781] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:24.948319 12306 ts_tablet_manager.cc:1397] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.008s
I20260807 07:59:24.948319 12339 raft_consensus.cc:482] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:24.953135 12339 raft_consensus.cc:3037] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.955075 12344 raft_consensus.cc:482] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:24.955528 12344 raft_consensus.cc:504] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.951982 12306 raft_consensus.cc:348] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.958068 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:24.960850 12025 raft_consensus.cc:2379] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52f278a746734ec1a96bce52e7bbba50 in current term 1: Already voted for candidate 31679f1015b349cf941d1a73f5786d47 in this term.
I20260807 07:59:24.961345 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47" is_pre_election: true
I20260807 07:59:24.960175 12344 leader_election.cc:288] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:24.962309 12339 raft_consensus.cc:504] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.959488 12311 raft_consensus.cc:2726] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:24.962735 12097 leader_election.cc:302] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50; no voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758
I20260807 07:59:24.960043 12313 raft_consensus.cc:2726] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:24.963831 12344 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.964179 12339 leader_election.cc:288] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:24.958177 12306 raft_consensus.cc:374] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:24.965380 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:24.965821 12306 raft_consensus.cc:729] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Initialized, Role: FOLLOWER
I20260807 07:59:24.966001 12025 raft_consensus.cc:3037] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.965818 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:24.966761 12140 raft_consensus.cc:3037] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:24.966670 12306 consensus_queue.cc:260] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:24.970598 12344 raft_consensus.cc:2726] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:24.971160 12306 ts_tablet_manager.cc:1428] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758: Time spent starting tablet: real 0.023s user 0.003s sys 0.003s
I20260807 07:59:24.972056 12306 tablet_bootstrap.cc:492] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758: Bootstrap starting.
I20260807 07:59:24.973156 12140 raft_consensus.cc:2454] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 1.
I20260807 07:59:24.974112 12197 leader_election.cc:302] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters:
I20260807 07:59:24.977928 12306 tablet_bootstrap.cc:654] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:24.978343 12311 raft_consensus.cc:2781] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:24.984723 12339 consensus_queue.cc:1035] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 07:59:24.987582 12025 raft_consensus.cc:2454] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 1.
I20260807 07:59:24.991017 12311 raft_consensus.cc:686] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Becoming Leader. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:24.995590 12311 consensus_queue.cc:237] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.000921 12306 tablet_bootstrap.cc:492] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758: No bootstrap required, opened a new log
I20260807 07:59:25.001492 12306 ts_tablet_manager.cc:1397] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.007s
I20260807 07:59:25.004570 12306 raft_consensus.cc:348] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.005481 12306 raft_consensus.cc:374] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:25.005831 12306 raft_consensus.cc:729] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Initialized, Role: FOLLOWER
I20260807 07:59:25.006553 12306 consensus_queue.cc:260] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.010578 12306 ts_tablet_manager.cc:1428] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260807 07:59:25.012508 11865 catalog_manager.cc:5495] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31679f1015b349cf941d1a73f5786d47 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "31679f1015b349cf941d1a73f5786d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:25.028779 12311 consensus_queue.cc:1035] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 07:59:25.064100 11950 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 07:59:25.066919 12140 raft_consensus.cc:3037] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:25.072865 12313 consensus_queue.cc:1035] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.083956 12400 consensus_queue.cc:1035] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 07:59:25.113672 12344 raft_consensus.cc:482] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:25.114146 12344 raft_consensus.cc:504] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.115619 12344 leader_election.cc:288] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:25.116462 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:25.116757 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47" is_pre_election: true
I20260807 07:59:25.116987 12025 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 0.
I20260807 07:59:25.117420 12249 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 0.
I20260807 07:59:25.117908 12096 leader_election.cc:302] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:25.118641 12344 raft_consensus.cc:2781] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:25.118996 12344 raft_consensus.cc:482] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:25.119259 12344 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:25.123335 12344 raft_consensus.cc:504] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.125115 12344 leader_election.cc:288] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:25.125507 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:25.125756 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:25.126096 12025 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:25.126284 12249 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:25.130509 12249 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 1.
I20260807 07:59:25.131454 12097 leader_election.cc:302] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters:
I20260807 07:59:25.132189 12344 raft_consensus.cc:2781] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:25.132124 12025 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 1.
I20260807 07:59:25.132895 12344 raft_consensus.cc:686] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:25.134383 12344 consensus_queue.cc:237] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.136405 12350 consensus_queue.cc:1035] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.143579 11869 catalog_manager.cc:5495] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 1 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:25.148578 12390 raft_consensus.cc:482] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:25.149044 12390 raft_consensus.cc:504] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.151100 12390 leader_election.cc:288] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:25.152987 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50" is_pre_election: true
I20260807 07:59:25.153645 12140 raft_consensus.cc:2454] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 0.
I20260807 07:59:25.154524 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758" is_pre_election: true
I20260807 07:59:25.155113 12025 raft_consensus.cc:2454] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 0.
I20260807 07:59:25.155951 12197 leader_election.cc:302] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters:
I20260807 07:59:25.157357 12390 raft_consensus.cc:2781] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:25.157761 12390 raft_consensus.cc:482] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:25.158038 12390 raft_consensus.cc:3037] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:25.157334 12344 consensus_queue.cc:1035] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260807 07:59:25.165593 12390 raft_consensus.cc:504] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.168411 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:25.168431 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:25.169015 12140 raft_consensus.cc:3037] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:25.169238 12026 raft_consensus.cc:3037] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:25.175118 12026 raft_consensus.cc:2454] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 1.
I20260807 07:59:25.175567 12140 raft_consensus.cc:2454] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 1.
I20260807 07:59:25.176165 12196 leader_election.cc:302] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:25.177033 12386 raft_consensus.cc:2781] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:25.177489 12386 raft_consensus.cc:686] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Becoming Leader. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:25.177599 12390 leader_election.cc:288] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 1 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:25.178347 12386 consensus_queue.cc:237] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.187479 11870 catalog_manager.cc:5495] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31679f1015b349cf941d1a73f5786d47 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "31679f1015b349cf941d1a73f5786d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:25.242983 12344 consensus_queue.cc:1035] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.256309 12344 consensus_queue.cc:1035] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.305979 12344 consensus_queue.cc:1035] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.311098 12344 consensus_queue.cc:1035] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.316893 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
mode: GRACEFUL
new_leader_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:36410
I20260807 07:59:25.317662 12140 raft_consensus.cc:595] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Received request to transfer leadership to TS e3cb34634f834e639999e07130187758
I20260807 07:59:25.332998 12026 raft_consensus.cc:3037] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:25.341142 12314 consensus_queue.cc:1035] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 07:59:25.345618 12314 consensus_queue.cc:1035] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.377283 12249 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
mode: GRACEFUL
new_leader_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:57982
I20260807 07:59:25.377931 12249 raft_consensus.cc:595] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Received request to transfer leadership to TS e3cb34634f834e639999e07130187758
I20260807 07:59:25.385082 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
mode: GRACEFUL
new_leader_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:36410
I20260807 07:59:25.385697 12140 raft_consensus.cc:595] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Received request to transfer leadership to TS e3cb34634f834e639999e07130187758
I20260807 07:59:25.392630 12249 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
mode: GRACEFUL
new_leader_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:57982
I20260807 07:59:25.393116 12249 raft_consensus.cc:595] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Received request to transfer leadership to TS 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:25.400332 12249 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
mode: GRACEFUL
new_leader_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:57982
I20260807 07:59:25.400887 12249 raft_consensus.cc:595] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Received request to transfer leadership to TS 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:25.405392 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
mode: GRACEFUL
new_leader_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:36410
I20260807 07:59:25.405982 12140 raft_consensus.cc:595] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Received request to transfer leadership to TS 31679f1015b349cf941d1a73f5786d47
I20260807 07:59:25.522469 12314 consensus_queue.cc:1035] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.531174 12314 raft_consensus.cc:979] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50: : Instructing follower 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:25.532109 12360 raft_consensus.cc:1067] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Signalling peer 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:25.534838 12250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:57978
I20260807 07:59:25.535414 12250 raft_consensus.cc:482] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:25.535776 12250 raft_consensus.cc:3037] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.543339 12350 consensus_queue.cc:1035] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.543295 12386 consensus_queue.cc:1035] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.546229 12250 raft_consensus.cc:504] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.549489 12250 leader_election.cc:288] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 2 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:25.550240 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:25.550844 12025 raft_consensus.cc:3037] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.551203 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9d77bf75b643218014f3d531b1cd29" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:25.552101 12140 raft_consensus.cc:3032] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:25.552455 12140 raft_consensus.cc:729] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:25.553398 12140 consensus_queue.cc:260] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.555267 12140 raft_consensus.cc:3037] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.559087 12025 raft_consensus.cc:2454] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 2.
I20260807 07:59:25.560748 12196 leader_election.cc:302] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:25.562480 12140 raft_consensus.cc:2454] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 2.
I20260807 07:59:25.563571 12386 raft_consensus.cc:2781] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:25.567631 12386 raft_consensus.cc:686] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 2 LEADER]: Becoming Leader. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:25.570634 12386 consensus_queue.cc:237] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.576145 12386 consensus_queue.cc:1035] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.581061 11870 catalog_manager.cc:5495] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 reported cstate change: term changed from 1 to 2, leader changed from 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66) to 31679f1015b349cf941d1a73f5786d47 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "31679f1015b349cf941d1a73f5786d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:25.585083 12386 raft_consensus.cc:979] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47: : Instructing follower 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:25.585513 12386 raft_consensus.cc:1067] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Signalling peer 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:25.587132 12140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36384
I20260807 07:59:25.587661 12140 raft_consensus.cc:482] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:25.588095 12140 raft_consensus.cc:3037] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.594383 12140 raft_consensus.cc:504] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.596606 12140 leader_election.cc:288] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 2 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:25.597081 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:25.597682 12026 raft_consensus.cc:3037] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.598104 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:25.598670 12249 raft_consensus.cc:3032] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:25.598994 12249 raft_consensus.cc:729] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:25.599702 12249 consensus_queue.cc:260] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.600896 12249 raft_consensus.cc:3037] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.605018 12026 raft_consensus.cc:2454] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 2.
I20260807 07:59:25.608279 12096 leader_election.cc:302] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:25.609112 12350 raft_consensus.cc:2781] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:25.609398 12386 raft_consensus.cc:979] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47: : Instructing follower e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:25.609448 12249 raft_consensus.cc:2454] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 2.
I20260807 07:59:25.609645 12350 raft_consensus.cc:686] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:25.609928 12386 raft_consensus.cc:1067] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Signalling peer e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:25.610502 12350 consensus_queue.cc:237] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.612941 12026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811"
dest_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:57538
I20260807 07:59:25.613461 12026 raft_consensus.cc:482] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:25.613831 12026 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.619606 12026 raft_consensus.cc:504] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.621198 12026 leader_election.cc:288] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 2 election: Requested vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:25.620556 11870 catalog_manager.cc:5495] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 1 to 2, leader changed from 31679f1015b349cf941d1a73f5786d47 (127.11.140.67) to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 2 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:25.622396 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:25.622757 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:25.622963 12140 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.623315 12249 raft_consensus.cc:3032] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:25.623651 12249 raft_consensus.cc:729] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:25.624385 12249 consensus_queue.cc:260] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.625586 12249 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.629130 12140 raft_consensus.cc:2454] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 2.
I20260807 07:59:25.630239 11984 leader_election.cc:302] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:25.631065 12400 raft_consensus.cc:2781] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:25.632081 12400 raft_consensus.cc:686] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Becoming Leader. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:25.632898 12400 consensus_queue.cc:237] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.633703 12249 raft_consensus.cc:2454] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 2.
I20260807 07:59:25.639768 11870 catalog_manager.cc:5495] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 reported cstate change: term changed from 1 to 2, leader changed from 31679f1015b349cf941d1a73f5786d47 (127.11.140.67) to e3cb34634f834e639999e07130187758 (127.11.140.65). New cstate: current_term: 2 leader_uuid: "e3cb34634f834e639999e07130187758" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:25.675268 12339 consensus_queue.cc:1035] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.681377 12390 raft_consensus.cc:979] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47: : Instructing follower 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:25.681818 12390 raft_consensus.cc:1067] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Signalling peer 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:25.683583 12139 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36384
I20260807 07:59:25.684134 12139 raft_consensus.cc:482] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:25.684484 12139 raft_consensus.cc:3037] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.687090 12390 consensus_queue.cc:1035] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.691752 12139 raft_consensus.cc:504] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.692817 12140 raft_consensus.cc:1226] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Rejecting Update request from peer 31679f1015b349cf941d1a73f5786d47 for earlier term 1. Current term is 2. Ops: []
I20260807 07:59:25.694615 12139 leader_election.cc:288] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 2 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:25.694536 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:25.697070 12390 consensus_queue.cc:1046] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:25.697641 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28366b85f02b466597fd0583d79b21fb" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:25.698249 12249 raft_consensus.cc:3032] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:25.698586 12249 raft_consensus.cc:729] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:25.699532 12249 consensus_queue.cc:260] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.700666 12249 raft_consensus.cc:3037] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.707506 12249 raft_consensus.cc:2454] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 2.
W20260807 07:59:25.707908 12386 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260807 07:59:25.709719 12097 leader_election.cc:302] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters: e3cb34634f834e639999e07130187758
I20260807 07:59:25.710562 12350 raft_consensus.cc:2781] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:25.711050 12350 raft_consensus.cc:686] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:25.711877 12350 consensus_queue.cc:237] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.719206 11870 catalog_manager.cc:5495] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 1 to 2, leader changed from 31679f1015b349cf941d1a73f5786d47 (127.11.140.67) to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 2 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:25.779795 12314 raft_consensus.cc:979] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50: : Instructing follower e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:25.780333 12350 raft_consensus.cc:1067] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Signalling peer e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:25.782207 12026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4"
dest_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:57542
I20260807 07:59:25.783187 12026 raft_consensus.cc:482] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:25.783598 12026 raft_consensus.cc:3037] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.788409 12026 raft_consensus.cc:504] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.790046 12026 leader_election.cc:288] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 2 election: Requested vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:25.791085 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:25.791329 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:25.791714 12140 raft_consensus.cc:3032] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:25.791989 12249 raft_consensus.cc:3037] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.792121 12140 raft_consensus.cc:729] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:25.793040 12140 consensus_queue.cc:260] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.794229 12140 raft_consensus.cc:3037] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.799265 12140 raft_consensus.cc:2454] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 2.
I20260807 07:59:25.799989 12249 raft_consensus.cc:2454] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 2.
I20260807 07:59:25.800275 11984 leader_election.cc:302] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:25.801123 12433 raft_consensus.cc:2781] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:25.801620 12433 raft_consensus.cc:686] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Becoming Leader. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:25.802515 12433 consensus_queue.cc:237] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:25.810709 11870 catalog_manager.cc:5495] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 reported cstate change: term changed from 1 to 2, leader changed from 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66) to e3cb34634f834e639999e07130187758 (127.11.140.65). New cstate: current_term: 2 leader_uuid: "e3cb34634f834e639999e07130187758" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:25.825232 12350 raft_consensus.cc:979] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50: : Instructing follower e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:25.825690 12314 raft_consensus.cc:1067] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Signalling peer e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:25.827011 12026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360"
dest_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:57542
I20260807 07:59:25.827482 12026 raft_consensus.cc:482] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:25.827869 12026 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.832975 12026 raft_consensus.cc:504] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:25.834743 12026 leader_election.cc:288] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 2 election: Requested vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:25.835652 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:25.835799 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:25.836302 12140 raft_consensus.cc:3032] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:25.836441 12249 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.836670 12140 raft_consensus.cc:729] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:25.837510 12140 consensus_queue.cc:260] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:25.838788 12140 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:25.841110 12249 raft_consensus.cc:2454] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 2.
I20260807 07:59:25.842095 11984 leader_election.cc:302] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:25.842783 12433 raft_consensus.cc:2781] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:25.843267 12433 raft_consensus.cc:686] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Becoming Leader. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:25.844007 12433 consensus_queue.cc:237] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:25.847635 12140 raft_consensus.cc:2454] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 2.
I20260807 07:59:25.851444 11870 catalog_manager.cc:5495] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 reported cstate change: term changed from 1 to 2, leader changed from 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66) to e3cb34634f834e639999e07130187758 (127.11.140.65). New cstate: current_term: 2 leader_uuid: "e3cb34634f834e639999e07130187758" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:26.025789 12140 raft_consensus.cc:1261] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.027107 12386 consensus_queue.cc:1035] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.034245 12026 raft_consensus.cc:1261] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.036793 12339 consensus_queue.cc:1035] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.037712 12026 raft_consensus.cc:1261] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.039362 12360 consensus_queue.cc:1035] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.047816 12249 raft_consensus.cc:1261] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.049266 12360 consensus_queue.cc:1035] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.194149 12249 raft_consensus.cc:1261] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.195277 12360 consensus_queue.cc:1035] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.196798 12249 raft_consensus.cc:1261] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.198089 12433 consensus_queue.cc:1035] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.204432 12025 raft_consensus.cc:3037] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:26.210701 12025 raft_consensus.cc:1261] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.211908 12360 consensus_queue.cc:1035] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.215543 12140 raft_consensus.cc:1261] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.216950 12440 consensus_queue.cc:1035] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.235262 12140 raft_consensus.cc:1261] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.236279 12433 consensus_queue.cc:1035] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.243213 12250 raft_consensus.cc:1261] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.245427 12440 consensus_queue.cc:1035] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.406069 12250 raft_consensus.cc:1261] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.407292 12440 consensus_queue.cc:1035] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:26.415782 12140 raft_consensus.cc:1261] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:26.418112 12434 consensus_queue.cc:1035] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:28.434834 11825 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260807 07:59:28.435726 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:28.458837 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b205118d55de47b185bf281b63fd2841"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
mode: GRACEFUL
new_leader_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:36418
I20260807 07:59:28.459352 12140 raft_consensus.cc:595] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Received request to transfer leadership to TS 31679f1015b349cf941d1a73f5786d47
I20260807 07:59:28.470606 12025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360"
dest_uuid: "e3cb34634f834e639999e07130187758"
mode: GRACEFUL
new_leader_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:57546
I20260807 07:59:28.471098 12025 raft_consensus.cc:595] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Received request to transfer leadership to TS 31679f1015b349cf941d1a73f5786d47
I20260807 07:59:28.472072 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260807 07:59:28.472539 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:28.733423 12350 raft_consensus.cc:979] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50: : Instructing follower 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:28.733909 12458 raft_consensus.cc:1067] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Signalling peer 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:28.735311 12250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b205118d55de47b185bf281b63fd2841"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:57978
I20260807 07:59:28.735780 12250 raft_consensus.cc:482] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:28.736060 12250 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:28.740293 12250 raft_consensus.cc:504] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:28.742195 12250 leader_election.cc:288] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 2 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:28.742604 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:28.742835 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:28.746913 12025 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:28.747138 12140 raft_consensus.cc:3032] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:28.747556 12140 raft_consensus.cc:729] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:28.748306 12140 consensus_queue.cc:260] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:28.749207 12140 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:28.751652 12025 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 2.
I20260807 07:59:28.752626 12196 leader_election.cc:302] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:28.753378 12450 raft_consensus.cc:2781] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:28.753927 12140 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 2.
I20260807 07:59:28.754812 12450 raft_consensus.cc:686] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 2 LEADER]: Becoming Leader. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:28.755695 12450 consensus_queue.cc:237] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:28.764293 11869 catalog_manager.cc:5495] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 reported cstate change: term changed from 1 to 2, leader changed from 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66) to 31679f1015b349cf941d1a73f5786d47 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "31679f1015b349cf941d1a73f5786d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:28.897292 12434 raft_consensus.cc:979] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758: : Instructing follower 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:28.897837 12470 raft_consensus.cc:1067] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Signalling peer 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:28.899298 12250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:57964
I20260807 07:59:28.899818 12250 raft_consensus.cc:482] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:28.900184 12250 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:28.904703 12250 raft_consensus.cc:504] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:28.906394 12250 leader_election.cc:288] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 3 election: Requested vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), e3cb34634f834e639999e07130187758 (127.11.140.65:43471)
I20260807 07:59:28.907069 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:28.907421 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:28.907657 12140 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:28.907986 12025 raft_consensus.cc:3032] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 07:59:28.908246 12025 raft_consensus.cc:729] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:28.908831 12025 consensus_queue.cc:260] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:28.909771 12025 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:28.912147 12140 raft_consensus.cc:2454] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 3.
I20260807 07:59:28.913080 12197 leader_election.cc:302] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters:
I20260807 07:59:28.913757 12450 raft_consensus.cc:2781] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:59:28.914232 12450 raft_consensus.cc:686] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 3 LEADER]: Becoming Leader. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:28.914351 12025 raft_consensus.cc:2454] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 3.
I20260807 07:59:28.915066 12450 consensus_queue.cc:237] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:28.921864 11869 catalog_manager.cc:5495] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 reported cstate change: term changed from 2 to 3, leader changed from e3cb34634f834e639999e07130187758 (127.11.140.65) to 31679f1015b349cf941d1a73f5786d47 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "31679f1015b349cf941d1a73f5786d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:29.223999 12025 raft_consensus.cc:1261] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:29.225179 12450 consensus_queue.cc:1035] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:29.234982 12140 raft_consensus.cc:1261] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:29.236649 12460 consensus_queue.cc:1035] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:29.303651 12025 raft_consensus.cc:1261] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:29.304731 12460 consensus_queue.cc:1035] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:29.309264 12140 raft_consensus.cc:1261] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:29.311134 12460 consensus_queue.cc:1035] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:29.473307 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:29.477883 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:29.478271 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:30.479095 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:30.483692 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:30.484087 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:31.484840 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:31.489466 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:31.490078 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:32.491225 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:32.496119 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:32.496443 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:33.524782 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:33.529204 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:33.529620 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260807 07:59:34.322490 11923 debug-util.cc:398] Leaking SignalData structure 0x7b0800060060 after lost signal to thread 11828
W20260807 07:59:34.323252 11923 debug-util.cc:398] Leaking SignalData structure 0x7b0800060740 after lost signal to thread 11928
W20260807 07:59:34.324009 11923 debug-util.cc:398] Leaking SignalData structure 0x7b0800113660 after lost signal to thread 12063
W20260807 07:59:34.324714 11923 debug-util.cc:398] Leaking SignalData structure 0x7b08001a9b60 after lost signal to thread 12177
W20260807 07:59:34.325369 11923 debug-util.cc:398] Leaking SignalData structure 0x7b08001d6a20 after lost signal to thread 12281
I20260807 07:59:34.530570 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:34.538551 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:34.539099 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:35.539873 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:35.545162 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:35.545596 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:36.548053 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:36.581246 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:36.582980 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:37.584110 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:37.588677 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:37.589092 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:38.614516 11825 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260807 07:59:38.619168 12250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
mode: GRACEFUL
new_leader_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:57982
I20260807 07:59:38.619869 12250 raft_consensus.cc:595] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 3 LEADER]: Received request to transfer leadership to TS 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:38.636737 12025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811"
dest_uuid: "e3cb34634f834e639999e07130187758"
mode: GRACEFUL
new_leader_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36694
I20260807 07:59:38.637180 12025 raft_consensus.cc:595] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Received request to transfer leadership to TS 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:38.641259 12025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d"
dest_uuid: "e3cb34634f834e639999e07130187758"
mode: GRACEFUL
new_leader_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36694
I20260807 07:59:38.641701 12025 raft_consensus.cc:595] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 1 LEADER]: Received request to transfer leadership to TS 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:38.646025 12025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4"
dest_uuid: "e3cb34634f834e639999e07130187758"
mode: GRACEFUL
new_leader_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36694
I20260807 07:59:38.646438 12025 raft_consensus.cc:595] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Received request to transfer leadership to TS 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:38.654651 12250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b205118d55de47b185bf281b63fd2841"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
mode: GRACEFUL
new_leader_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:57982
I20260807 07:59:38.655037 12250 raft_consensus.cc:595] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 2 LEADER]: Received request to transfer leadership to TS 52f278a746734ec1a96bce52e7bbba50
I20260807 07:59:38.809978 12493 raft_consensus.cc:979] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47: : Instructing follower 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:38.810636 12504 raft_consensus.cc:1067] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 3 LEADER]: Signalling peer 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:38.812176 12140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36384
I20260807 07:59:38.812727 12140 raft_consensus.cc:482] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:38.813089 12140 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Advancing to term 4
I20260807 07:59:38.817747 12140 raft_consensus.cc:504] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:38.819344 12140 leader_election.cc:288] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 4 election: Requested vote from peers 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147), e3cb34634f834e639999e07130187758 (127.11.140.65:43471)
I20260807 07:59:38.820259 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:38.820596 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f442cf60f9a64b8398f1fa3db587c360" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:38.820868 12250 raft_consensus.cc:3032] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 07:59:38.821158 12025 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Advancing to term 4
I20260807 07:59:38.821292 12250 raft_consensus.cc:729] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:38.822178 12250 consensus_queue.cc:260] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:38.823355 12250 raft_consensus.cc:3037] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Advancing to term 4
I20260807 07:59:38.825668 12025 raft_consensus.cc:2454] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 4.
I20260807 07:59:38.826736 12096 leader_election.cc:302] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:38.827500 12495 raft_consensus.cc:2781] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 07:59:38.827623 12250 raft_consensus.cc:2454] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 4.
I20260807 07:59:38.828511 12495 raft_consensus.cc:686] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 4 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:38.829551 12495 consensus_queue.cc:237] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } }
I20260807 07:59:38.838586 11865 catalog_manager.cc:5495] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 3 to 4, leader changed from 31679f1015b349cf941d1a73f5786d47 (127.11.140.67) to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 4 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:38.861332 12493 raft_consensus.cc:979] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47: : Instructing follower 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:38.861819 12504 raft_consensus.cc:1067] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 2 LEADER]: Signalling peer 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:38.863440 12140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b205118d55de47b185bf281b63fd2841"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36384
I20260807 07:59:38.864037 12140 raft_consensus.cc:482] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:38.864377 12140 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:38.869027 12140 raft_consensus.cc:504] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:38.870405 12140 leader_election.cc:288] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 3 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:38.871042 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:38.871595 12025 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:38.871632 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:38.872164 12250 raft_consensus.cc:3032] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 07:59:38.872404 12250 raft_consensus.cc:729] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:38.873011 12250 consensus_queue.cc:260] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:38.873835 12250 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:38.875435 12025 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 3.
I20260807 07:59:38.876230 12096 leader_election.cc:302] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:38.876881 12495 raft_consensus.cc:2781] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:59:38.877315 12495 raft_consensus.cc:686] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:38.878330 12250 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 3.
I20260807 07:59:38.878170 12495 consensus_queue.cc:237] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:38.885686 11865 catalog_manager.cc:5495] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 2 to 3, leader changed from 31679f1015b349cf941d1a73f5786d47 (127.11.140.67) to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 3 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:38.896140 12470 raft_consensus.cc:979] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758: : Instructing follower 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:38.896644 12515 raft_consensus.cc:1067] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Signalling peer 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:38.898028 12140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36398
I20260807 07:59:38.898485 12140 raft_consensus.cc:482] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:38.898802 12140 raft_consensus.cc:3037] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:38.902613 12140 raft_consensus.cc:504] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:38.903961 12140 leader_election.cc:288] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 3 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:38.904654 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:38.904947 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee27aa48f7045a9b753c390eb26aeb4" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:38.905174 12025 raft_consensus.cc:3032] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 07:59:38.905431 12250 raft_consensus.cc:3037] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:38.905450 12025 raft_consensus.cc:729] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:38.906252 12025 consensus_queue.cc:260] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:38.907378 12025 raft_consensus.cc:3037] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:38.911193 12250 raft_consensus.cc:2454] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 3.
I20260807 07:59:38.912207 12097 leader_election.cc:302] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters:
I20260807 07:59:38.912942 12495 raft_consensus.cc:2781] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:59:38.913276 12025 raft_consensus.cc:2454] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 3.
I20260807 07:59:38.913338 12495 raft_consensus.cc:686] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:38.914335 12495 consensus_queue.cc:237] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:38.920827 11865 catalog_manager.cc:5495] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 2 to 3, leader changed from e3cb34634f834e639999e07130187758 (127.11.140.65) to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 3 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:38.925562 12470 raft_consensus.cc:979] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758: : Instructing follower 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:38.925989 12515 raft_consensus.cc:1067] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Signalling peer 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:38.927445 12140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36398
I20260807 07:59:38.927857 12140 raft_consensus.cc:482] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:38.928158 12140 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:38.932116 12140 raft_consensus.cc:504] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:38.933259 12140 leader_election.cc:288] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 3 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:38.934010 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:38.934280 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:38.934450 12025 raft_consensus.cc:3032] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 07:59:38.934789 12250 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:38.934790 12025 raft_consensus.cc:729] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:38.935559 12025 consensus_queue.cc:260] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:38.936394 12025 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:38.939090 12250 raft_consensus.cc:2454] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 3.
I20260807 07:59:38.939913 12097 leader_election.cc:302] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters:
I20260807 07:59:38.940335 12025 raft_consensus.cc:2454] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 3.
I20260807 07:59:38.940680 12495 raft_consensus.cc:2781] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:59:38.941108 12495 raft_consensus.cc:686] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:38.941926 12495 consensus_queue.cc:237] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:38.948525 11865 catalog_manager.cc:5495] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 2 to 3, leader changed from e3cb34634f834e639999e07130187758 (127.11.140.65) to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 3 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:39.232165 12470 raft_consensus.cc:979] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758: : Instructing follower 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:39.232592 12515 raft_consensus.cc:1067] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 1 LEADER]: Signalling peer 52f278a746734ec1a96bce52e7bbba50 to start an election
I20260807 07:59:39.233793 12140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
from {username='slave'} at 127.0.0.1:36398
I20260807 07:59:39.234246 12140 raft_consensus.cc:482] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:39.234526 12140 raft_consensus.cc:3037] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:39.239392 12140 raft_consensus.cc:504] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:39.240906 12140 leader_election.cc:288] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 2 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:39.241704 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:39.241895 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d" candidate_uuid: "52f278a746734ec1a96bce52e7bbba50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:39.242238 12025 raft_consensus.cc:3032] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:39.242390 12250 raft_consensus.cc:3037] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:39.242596 12025 raft_consensus.cc:729] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:39.243383 12025 consensus_queue.cc:260] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:39.244306 12025 raft_consensus.cc:3037] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:39.247124 12250 raft_consensus.cc:2454] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 2.
I20260807 07:59:39.247983 12097 leader_election.cc:302] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, 52f278a746734ec1a96bce52e7bbba50; no voters:
I20260807 07:59:39.248569 12495 raft_consensus.cc:2781] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:39.248701 12025 raft_consensus.cc:2454] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f278a746734ec1a96bce52e7bbba50 in term 2.
I20260807 07:59:39.248986 12495 raft_consensus.cc:686] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Becoming Leader. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:39.250416 12495 consensus_queue.cc:237] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:39.257078 11865 catalog_manager.cc:5495] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 reported cstate change: term changed from 1 to 2, leader changed from e3cb34634f834e639999e07130187758 (127.11.140.65) to 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66). New cstate: current_term: 2 leader_uuid: "52f278a746734ec1a96bce52e7bbba50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:39.282342 12250 raft_consensus.cc:1261] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 07:59:39.283365 12495 consensus_queue.cc:1035] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 07:59:39.292451 12025 raft_consensus.cc:1261] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 07:59:39.293668 12506 consensus_queue.cc:1035] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 07:59:39.298336 12250 raft_consensus.cc:1261] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:39.299757 12532 consensus_queue.cc:1035] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 07:59:39.308293 12025 raft_consensus.cc:1261] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:39.310771 12026 raft_consensus.cc:1261] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:39.312852 12495 consensus_queue.cc:1035] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:39.314097 12495 consensus_queue.cc:1035] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:39.324823 12250 raft_consensus.cc:1261] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:39.326088 12495 consensus_queue.cc:1035] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:39.343169 12026 raft_consensus.cc:1261] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:39.344187 12495 consensus_queue.cc:1035] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:39.352087 12250 raft_consensus.cc:1261] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:39.355520 12545 consensus_queue.cc:1035] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 07:59:39.710662 12026 raft_consensus.cc:1261] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:39.711795 12545 consensus_queue.cc:1035] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:39.720121 12250 raft_consensus.cc:1261] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Refusing update from remote peer 52f278a746734ec1a96bce52e7bbba50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:39.721928 12532 consensus_queue.cc:1035] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:41.688896 11825 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260807 07:59:41.689746 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:41.697219 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
mode: GRACEFUL
new_leader_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:36418
I20260807 07:59:41.698057 12140 raft_consensus.cc:595] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Received request to transfer leadership to TS 31679f1015b349cf941d1a73f5786d47
I20260807 07:59:41.700380 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
mode: GRACEFUL
new_leader_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:36418
I20260807 07:59:41.700824 12140 raft_consensus.cc:595] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Received request to transfer leadership to TS 31679f1015b349cf941d1a73f5786d47
I20260807 07:59:41.703158 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b205118d55de47b185bf281b63fd2841"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
mode: GRACEFUL
new_leader_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:36418
I20260807 07:59:41.703644 12140 raft_consensus.cc:595] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Received request to transfer leadership to TS e3cb34634f834e639999e07130187758
I20260807 07:59:41.705936 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c972c80c939942feb0b61190b3d16449"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
mode: GRACEFUL
new_leader_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:36418
I20260807 07:59:41.706355 12140 raft_consensus.cc:595] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Received request to transfer leadership to TS e3cb34634f834e639999e07130187758
I20260807 07:59:41.708294 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811"
dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
mode: GRACEFUL
new_leader_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:36418
I20260807 07:59:41.708724 12140 raft_consensus.cc:595] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Received request to transfer leadership to TS e3cb34634f834e639999e07130187758
I20260807 07:59:41.709645 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 07:59:41.710052 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:41.774042 12555 raft_consensus.cc:979] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50: : Instructing follower 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:41.774729 12532 raft_consensus.cc:1067] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Signalling peer 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:41.776162 12250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:57978
I20260807 07:59:41.776559 12250 raft_consensus.cc:482] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:41.776819 12250 raft_consensus.cc:3037] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:41.781813 12250 raft_consensus.cc:504] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.783756 12250 leader_election.cc:288] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 3 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:41.784586 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:41.784811 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfb176244ad4dddae71d095b33c7326" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:41.785264 12026 raft_consensus.cc:3037] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:41.785362 12140 raft_consensus.cc:3032] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 07:59:41.785763 12140 raft_consensus.cc:729] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:41.786312 12140 consensus_queue.cc:260] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.787330 12140 raft_consensus.cc:3037] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:41.790205 12026 raft_consensus.cc:2454] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 3.
I20260807 07:59:41.791395 12196 leader_election.cc:302] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:41.792089 12140 raft_consensus.cc:2454] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 3.
I20260807 07:59:41.792349 12504 raft_consensus.cc:2781] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:59:41.793640 12504 raft_consensus.cc:686] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 3 LEADER]: Becoming Leader. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:41.794543 12504 consensus_queue.cc:237] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.801492 11870 catalog_manager.cc:5495] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 reported cstate change: term changed from 2 to 3, leader changed from 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66) to 31679f1015b349cf941d1a73f5786d47 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "31679f1015b349cf941d1a73f5786d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:41.870915 12532 raft_consensus.cc:979] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50: : Instructing follower 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:41.871364 12555 raft_consensus.cc:1067] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Signalling peer 31679f1015b349cf941d1a73f5786d47 to start an election
I20260807 07:59:41.872692 12250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d"
dest_uuid: "31679f1015b349cf941d1a73f5786d47"
from {username='slave'} at 127.0.0.1:57978
I20260807 07:59:41.873144 12250 raft_consensus.cc:482] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:41.873440 12250 raft_consensus.cc:3037] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:41.879283 12250 raft_consensus.cc:504] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.880705 12250 leader_election.cc:288] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 3 election: Requested vote from peers e3cb34634f834e639999e07130187758 (127.11.140.65:43471), 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089)
I20260807 07:59:41.881649 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:41.881762 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289aa3803f5a417e81efe7b311171d6d" candidate_uuid: "31679f1015b349cf941d1a73f5786d47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3cb34634f834e639999e07130187758"
I20260807 07:59:41.882082 12140 raft_consensus.cc:3032] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 07:59:41.882330 12026 raft_consensus.cc:3037] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:41.882372 12140 raft_consensus.cc:729] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:41.883214 12140 consensus_queue.cc:260] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.884101 12140 raft_consensus.cc:3037] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:41.886941 12026 raft_consensus.cc:2454] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 3.
I20260807 07:59:41.887876 12196 leader_election.cc:302] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:41.888170 12140 raft_consensus.cc:2454] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31679f1015b349cf941d1a73f5786d47 in term 3.
I20260807 07:59:41.888622 12504 raft_consensus.cc:2781] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:59:41.889122 12504 raft_consensus.cc:686] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 3 LEADER]: Becoming Leader. State: Replica: 31679f1015b349cf941d1a73f5786d47, State: Running, Role: LEADER
I20260807 07:59:41.889879 12504 consensus_queue.cc:237] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.896245 11870 catalog_manager.cc:5495] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 reported cstate change: term changed from 2 to 3, leader changed from 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66) to 31679f1015b349cf941d1a73f5786d47 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "31679f1015b349cf941d1a73f5786d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:41.901644 12532 raft_consensus.cc:979] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50: : Instructing follower e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:41.902043 12555 raft_consensus.cc:1067] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Signalling peer e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:41.903582 12026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811"
dest_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:57542
I20260807 07:59:41.904755 12026 raft_consensus.cc:482] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:41.905148 12026 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Advancing to term 4
I20260807 07:59:41.905656 12555 raft_consensus.cc:979] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50: : Instructing follower e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:41.906083 12532 raft_consensus.cc:1067] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Signalling peer e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:41.907371 12025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b205118d55de47b185bf281b63fd2841"
dest_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:57542
I20260807 07:59:41.907771 12025 raft_consensus.cc:482] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:41.908007 12025 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Advancing to term 4
I20260807 07:59:41.910046 12026 raft_consensus.cc:504] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.911839 12026 leader_election.cc:288] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 4 election: Requested vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:41.912837 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:41.912768 12025 raft_consensus.cc:504] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.913159 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bc59ebb68049018d4a79e610014811" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:41.913462 12140 raft_consensus.cc:3032] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 07:59:41.913810 12140 raft_consensus.cc:729] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:41.914676 12025 leader_election.cc:288] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 4 election: Requested vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:41.914525 12140 consensus_queue.cc:260] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.915648 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:41.915807 12140 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Advancing to term 4
I20260807 07:59:41.915829 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b205118d55de47b185bf281b63fd2841" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:41.916235 12139 raft_consensus.cc:3032] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 07:59:41.916359 12249 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Advancing to term 4
I20260807 07:59:41.916554 12139 raft_consensus.cc:729] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:41.913728 12250 raft_consensus.cc:3037] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Advancing to term 4
I20260807 07:59:41.917367 12139 consensus_queue.cc:260] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.918733 12139 raft_consensus.cc:3037] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Advancing to term 4
I20260807 07:59:41.922019 12140 raft_consensus.cc:2454] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 4.
I20260807 07:59:41.922879 12249 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 4.
I20260807 07:59:41.923417 11984 leader_election.cc:302] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f278a746734ec1a96bce52e7bbba50, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:41.923857 12250 raft_consensus.cc:2454] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 4.
I20260807 07:59:41.925241 12139 raft_consensus.cc:2454] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 4.
I20260807 07:59:41.925328 12572 raft_consensus.cc:2781] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 07:59:41.925313 11984 leader_election.cc:302] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:41.927470 12573 raft_consensus.cc:2781] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 07:59:41.927934 12573 raft_consensus.cc:686] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 4 LEADER]: Becoming Leader. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:41.928284 12572 raft_consensus.cc:686] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 4 LEADER]: Becoming Leader. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:41.928879 12573 consensus_queue.cc:237] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.929029 12572 consensus_queue.cc:237] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:41.936594 11870 catalog_manager.cc:5495] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 reported cstate change: term changed from 3 to 4, leader changed from 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66) to e3cb34634f834e639999e07130187758 (127.11.140.65). New cstate: current_term: 4 leader_uuid: "e3cb34634f834e639999e07130187758" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:41.947059 11870 catalog_manager.cc:5495] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 reported cstate change: term changed from 3 to 4, leader changed from 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66) to e3cb34634f834e639999e07130187758 (127.11.140.65). New cstate: current_term: 4 leader_uuid: "e3cb34634f834e639999e07130187758" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:42.026320 12555 raft_consensus.cc:979] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50: : Instructing follower e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:42.026743 12532 raft_consensus.cc:1067] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Signalling peer e3cb34634f834e639999e07130187758 to start an election
I20260807 07:59:42.027906 12025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c972c80c939942feb0b61190b3d16449"
dest_uuid: "e3cb34634f834e639999e07130187758"
from {username='slave'} at 127.0.0.1:57542
I20260807 07:59:42.028357 12025 raft_consensus.cc:482] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:42.028630 12025 raft_consensus.cc:3037] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:42.032876 12025 raft_consensus.cc:504] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:42.034142 12025 leader_election.cc:288] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 2 election: Requested vote from peers 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089), 31679f1015b349cf941d1a73f5786d47 (127.11.140.67:33147)
I20260807 07:59:42.034996 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c972c80c939942feb0b61190b3d16449" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52f278a746734ec1a96bce52e7bbba50"
I20260807 07:59:42.035156 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c972c80c939942feb0b61190b3d16449" candidate_uuid: "e3cb34634f834e639999e07130187758" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31679f1015b349cf941d1a73f5786d47"
I20260807 07:59:42.035487 12139 raft_consensus.cc:3032] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:42.035714 12250 raft_consensus.cc:3037] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:42.035836 12139 raft_consensus.cc:729] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52f278a746734ec1a96bce52e7bbba50, State: Running, Role: LEADER
I20260807 07:59:42.036705 12139 consensus_queue.cc:260] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:42.037949 12139 raft_consensus.cc:3037] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:42.040767 12250 raft_consensus.cc:2454] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 2.
I20260807 07:59:42.041589 11984 leader_election.cc:302] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31679f1015b349cf941d1a73f5786d47, e3cb34634f834e639999e07130187758; no voters:
I20260807 07:59:42.042201 12573 raft_consensus.cc:2781] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:42.042594 12573 raft_consensus.cc:686] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Becoming Leader. State: Replica: e3cb34634f834e639999e07130187758, State: Running, Role: LEADER
I20260807 07:59:42.042733 12139 raft_consensus.cc:2454] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cb34634f834e639999e07130187758 in term 2.
I20260807 07:59:42.043414 12573 consensus_queue.cc:237] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } }
I20260807 07:59:42.049824 11870 catalog_manager.cc:5495] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 reported cstate change: term changed from 1 to 2, leader changed from 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66) to e3cb34634f834e639999e07130187758 (127.11.140.65). New cstate: current_term: 2 leader_uuid: "e3cb34634f834e639999e07130187758" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:42.293160 12139 raft_consensus.cc:1261] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:42.294461 12504 consensus_queue.cc:1035] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:42.304454 12025 raft_consensus.cc:1261] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:42.306119 12583 consensus_queue.cc:1035] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 07:59:42.342937 12139 raft_consensus.cc:1261] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 07:59:42.344153 12573 consensus_queue.cc:1035] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 07:59:42.352757 12250 raft_consensus.cc:1261] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 07:59:42.354162 12573 consensus_queue.cc:1035] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 07:59:42.430259 12139 raft_consensus.cc:1261] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:42.431459 12583 consensus_queue.cc:1035] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:42.440048 12025 raft_consensus.cc:1261] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Refusing update from remote peer 31679f1015b349cf941d1a73f5786d47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:42.442245 12557 consensus_queue.cc:1035] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cb34634f834e639999e07130187758" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 07:59:42.460556 12250 raft_consensus.cc:1261] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:42.461661 12574 consensus_queue.cc:1035] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:42.470644 12139 raft_consensus.cc:1261] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:42.471884 12574 consensus_queue.cc:1035] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:42.506008 12250 raft_consensus.cc:1261] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 07:59:42.507123 12573 consensus_queue.cc:1035] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31679f1015b349cf941d1a73f5786d47" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 07:59:42.515763 12139 raft_consensus.cc:1261] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Refusing update from remote peer e3cb34634f834e639999e07130187758: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 07:59:42.517869 12572 consensus_queue.cc:1035] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f278a746734ec1a96bce52e7bbba50" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 07:59:42.710832 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:42.715440 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:42.715797 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:43.716738 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:43.721932 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:43.722272 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:44.722951 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:44.727519 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:44.727847 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:45.728523 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:45.733242 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:45.733783 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:46.734555 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:46.739050 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:46.739424 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:47.740252 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:47.745007 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:47.745399 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:48.746222 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:48.752825 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:48.753221 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:49.753989 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:49.758533 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:49.758836 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:50.759634 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:50.764221 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 07:59:50.764540 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:51.792449 11825 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260807 07:59:51.793278 11825 tablet_server.cc:178] TabletServer@127.11.140.65:0 shutting down...
W20260807 07:59:51.797150 12196 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260807 07:59:51.808537 12196 consensus_peers.cc:487] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 -> Peer e3cb34634f834e639999e07130187758 (127.11.140.65:43471): Couldn't send request to peer e3cb34634f834e639999e07130187758. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260807 07:59:51.835018 11825 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 07:59:51.835933 11825 tablet_replica.cc:330] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758: stopping tablet replica
I20260807 07:59:51.837379 11825 raft_consensus.cc:2229] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 4 LEADER]: Raft consensus shutting down.
I20260807 07:59:51.838603 11825 raft_consensus.cc:2258] T b205118d55de47b185bf281b63fd2841 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:51.840790 11825 tablet_replica.cc:330] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758: stopping tablet replica
I20260807 07:59:51.841483 11825 raft_consensus.cc:2229] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:51.842175 11825 raft_consensus.cc:2258] T 28366b85f02b466597fd0583d79b21fb P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:51.844456 11825 tablet_replica.cc:330] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758: stopping tablet replica
I20260807 07:59:51.845078 11825 raft_consensus.cc:2229] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:51.845630 11825 raft_consensus.cc:2258] T 0dfb176244ad4dddae71d095b33c7326 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:51.847433 11825 tablet_replica.cc:330] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758: stopping tablet replica
I20260807 07:59:51.847915 11825 raft_consensus.cc:2229] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 4 LEADER]: Raft consensus shutting down.
I20260807 07:59:51.848907 11825 raft_consensus.cc:2258] T f5bc59ebb68049018d4a79e610014811 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:51.850703 11825 tablet_replica.cc:330] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758: stopping tablet replica
I20260807 07:59:51.851164 11825 raft_consensus.cc:2229] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 2 LEADER]: Raft consensus shutting down.
I20260807 07:59:51.852068 11825 raft_consensus.cc:2258] T c972c80c939942feb0b61190b3d16449 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:51.853714 11825 tablet_replica.cc:330] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758: stopping tablet replica
I20260807 07:59:51.854195 11825 raft_consensus.cc:2229] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:51.854581 11825 raft_consensus.cc:2258] T 8b9d77bf75b643218014f3d531b1cd29 P e3cb34634f834e639999e07130187758 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:51.856006 11825 tablet_replica.cc:330] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758: stopping tablet replica
I20260807 07:59:51.856441 11825 raft_consensus.cc:2229] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:51.856892 11825 raft_consensus.cc:2258] T 289aa3803f5a417e81efe7b311171d6d P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:51.858676 11825 tablet_replica.cc:330] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758: stopping tablet replica
I20260807 07:59:51.859094 11825 raft_consensus.cc:2229] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:51.859428 11825 raft_consensus.cc:2258] T 4ee27aa48f7045a9b753c390eb26aeb4 P e3cb34634f834e639999e07130187758 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:51.860960 11825 tablet_replica.cc:330] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758: stopping tablet replica
I20260807 07:59:51.861353 11825 raft_consensus.cc:2229] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:51.861835 11825 raft_consensus.cc:2258] T f442cf60f9a64b8398f1fa3db587c360 P e3cb34634f834e639999e07130187758 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:51.942644 11825 tablet_server.cc:194] TabletServer@127.11.140.65:0 shutdown complete.
I20260807 07:59:51.965368 11825 tablet_server.cc:178] TabletServer@127.11.140.66:0 shutting down...
I20260807 07:59:51.996655 11825 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 07:59:51.997555 11825 tablet_replica.cc:330] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50: stopping tablet replica
I20260807 07:59:51.998405 11825 raft_consensus.cc:2229] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:51.999167 11825 raft_consensus.cc:2258] T b205118d55de47b185bf281b63fd2841 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.001780 11825 tablet_replica.cc:330] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50: stopping tablet replica
W20260807 07:59:52.002395 12197 consensus_peers.cc:487] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 -> Peer 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089): Couldn't send request to peer 52f278a746734ec1a96bce52e7bbba50. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260807 07:59:52.002713 11825 raft_consensus.cc:2229] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.003290 11825 raft_consensus.cc:2258] T 0dfb176244ad4dddae71d095b33c7326 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260807 07:59:52.004458 12096 consensus_peers.cc:487] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 -> Peer e3cb34634f834e639999e07130187758 (127.11.140.65:43471): Couldn't send request to peer e3cb34634f834e639999e07130187758. Status: Network error: Client connection negotiation failed: client connection to 127.11.140.65:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 07:59:52.005378 11825 tablet_replica.cc:330] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50: stopping tablet replica
I20260807 07:59:52.005932 11825 raft_consensus.cc:2229] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.006343 11825 raft_consensus.cc:2258] T f5bc59ebb68049018d4a79e610014811 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.007896 11825 tablet_replica.cc:330] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50: stopping tablet replica
I20260807 07:59:52.008337 11825 raft_consensus.cc:2229] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 2 LEADER]: Raft consensus shutting down.
I20260807 07:59:52.009063 11825 raft_consensus.cc:2258] T 28366b85f02b466597fd0583d79b21fb P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.010792 11825 tablet_replica.cc:330] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50: stopping tablet replica
I20260807 07:59:52.011299 11825 raft_consensus.cc:2229] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.011744 11825 raft_consensus.cc:2258] T 289aa3803f5a417e81efe7b311171d6d P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.013274 11825 tablet_replica.cc:330] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50: stopping tablet replica
I20260807 07:59:52.013775 11825 raft_consensus.cc:2229] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.014209 11825 raft_consensus.cc:2258] T c972c80c939942feb0b61190b3d16449 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.015699 11825 tablet_replica.cc:330] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50: stopping tablet replica
I20260807 07:59:52.016147 11825 raft_consensus.cc:2229] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.016618 11825 raft_consensus.cc:2258] T 8b9d77bf75b643218014f3d531b1cd29 P 52f278a746734ec1a96bce52e7bbba50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.018175 11825 tablet_replica.cc:330] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50: stopping tablet replica
I20260807 07:59:52.018635 11825 raft_consensus.cc:2229] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 3 LEADER]: Raft consensus shutting down.
I20260807 07:59:52.019388 11825 raft_consensus.cc:2258] T 4ee27aa48f7045a9b753c390eb26aeb4 P 52f278a746734ec1a96bce52e7bbba50 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.021044 11825 tablet_replica.cc:330] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50: stopping tablet replica
I20260807 07:59:52.021472 11825 raft_consensus.cc:2229] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 4 LEADER]: Raft consensus shutting down.
I20260807 07:59:52.022250 11825 raft_consensus.cc:2258] T f442cf60f9a64b8398f1fa3db587c360 P 52f278a746734ec1a96bce52e7bbba50 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260807 07:59:52.041011 12197 consensus_peers.cc:487] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 -> Peer 52f278a746734ec1a96bce52e7bbba50 (127.11.140.66:33089): Couldn't send request to peer 52f278a746734ec1a96bce52e7bbba50. Status: Network error: Client connection negotiation failed: client connection to 127.11.140.66:33089: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 07:59:52.050117 11825 tablet_server.cc:194] TabletServer@127.11.140.66:0 shutdown complete.
I20260807 07:59:52.067706 11825 tablet_server.cc:178] TabletServer@127.11.140.67:0 shutting down...
W20260807 07:59:52.078271 12196 consensus_peers.cc:487] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 -> Peer e3cb34634f834e639999e07130187758 (127.11.140.65:43471): Couldn't send request to peer e3cb34634f834e639999e07130187758. Status: Network error: Client connection negotiation failed: client connection to 127.11.140.65:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 07:59:52.095920 11825 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 07:59:52.096748 11825 tablet_replica.cc:330] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47: stopping tablet replica
I20260807 07:59:52.097450 11825 raft_consensus.cc:2229] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.098030 11825 raft_consensus.cc:2258] T 4ee27aa48f7045a9b753c390eb26aeb4 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.099781 11825 tablet_replica.cc:330] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47: stopping tablet replica
I20260807 07:59:52.100337 11825 raft_consensus.cc:2229] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.100755 11825 raft_consensus.cc:2258] T f442cf60f9a64b8398f1fa3db587c360 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.102399 11825 tablet_replica.cc:330] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47: stopping tablet replica
I20260807 07:59:52.102882 11825 raft_consensus.cc:2229] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.103291 11825 raft_consensus.cc:2258] T 28366b85f02b466597fd0583d79b21fb P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.104725 11825 tablet_replica.cc:330] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47: stopping tablet replica
I20260807 07:59:52.105180 11825 raft_consensus.cc:2229] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 3 LEADER]: Raft consensus shutting down.
I20260807 07:59:52.105974 11825 raft_consensus.cc:2258] T 0dfb176244ad4dddae71d095b33c7326 P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.107511 11825 tablet_replica.cc:330] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47: stopping tablet replica
I20260807 07:59:52.108014 11825 raft_consensus.cc:2229] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.108487 11825 raft_consensus.cc:2258] T f5bc59ebb68049018d4a79e610014811 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.110082 11825 tablet_replica.cc:330] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47: stopping tablet replica
I20260807 07:59:52.110555 11825 raft_consensus.cc:2229] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.111027 11825 raft_consensus.cc:2258] T c972c80c939942feb0b61190b3d16449 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.112531 11825 tablet_replica.cc:330] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47: stopping tablet replica
I20260807 07:59:52.112984 11825 raft_consensus.cc:2229] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 2 LEADER]: Raft consensus shutting down.
I20260807 07:59:52.113785 11825 raft_consensus.cc:2258] T 8b9d77bf75b643218014f3d531b1cd29 P 31679f1015b349cf941d1a73f5786d47 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.115531 11825 tablet_replica.cc:330] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47: stopping tablet replica
I20260807 07:59:52.116004 11825 raft_consensus.cc:2229] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 3 LEADER]: Raft consensus shutting down.
I20260807 07:59:52.116816 11825 raft_consensus.cc:2258] T 289aa3803f5a417e81efe7b311171d6d P 31679f1015b349cf941d1a73f5786d47 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.118918 11825 tablet_replica.cc:330] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47: stopping tablet replica
I20260807 07:59:52.119428 11825 raft_consensus.cc:2229] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 07:59:52.119901 11825 raft_consensus.cc:2258] T b205118d55de47b185bf281b63fd2841 P 31679f1015b349cf941d1a73f5786d47 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.148005 11825 tablet_server.cc:194] TabletServer@127.11.140.67:0 shutdown complete.
I20260807 07:59:52.164707 11825 master.cc:537] Master@127.11.140.126:39103 shutting down...
I20260807 07:59:52.465456 11825 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 1 LEADER]: Raft consensus shutting down.
I20260807 07:59:52.466084 11825 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 07:59:52.466370 11825 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7e69eef013a14ace8857d22c2b2461fd: stopping tablet replica
I20260807 07:59:52.501881 11825 master.cc:552] Master@127.11.140.126:39103 shutdown complete.
31679f1015b349cf941d1a73f5786d47 1
52f278a746734ec1a96bce52e7bbba50 4
e3cb34634f834e639999e07130187758 4
31679f1015b349cf941d1a73f5786d47 3
52f278a746734ec1a96bce52e7bbba50 3
e3cb34634f834e639999e07130187758 3
31679f1015b349cf941d1a73f5786d47 1
52f278a746734ec1a96bce52e7bbba50 8
e3cb34634f834e639999e07130187758 0
31679f1015b349cf941d1a73f5786d47 3
52f278a746734ec1a96bce52e7bbba50 3
e3cb34634f834e639999e07130187758 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30979 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260807 07:59:52.542168 11825 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.140.126:46249
I20260807 07:59:52.543175 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:59:52.548846 12622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:52.549585 12624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:52.550098 12626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:52.550388 11825 server_base.cc:1034] running on GCE node
I20260807 07:59:52.551523 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 07:59:52.551749 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 07:59:52.551900 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089592551874 us; error 0 us; skew 500 ppm
I20260807 07:59:52.552404 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:59:52.558272 11825 webserver.cc:460] Webserver started at http://127.11.140.126:42603/ using document root <none> and password file <none>
I20260807 07:59:52.558744 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 07:59:52.558931 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:59:52.559221 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:59:52.560410 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/master-0-root/instance:
uuid: "e484dc6c578f4ea1acf844cb0e662be4"
format_stamp: "Formatted at 2026-08-07 07:59:52 on dist-test-slave-zr1t"
I20260807 07:59:52.564706 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 07:59:52.567780 12634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:52.568414 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 07:59:52.568688 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/master-0-root
uuid: "e484dc6c578f4ea1acf844cb0e662be4"
format_stamp: "Formatted at 2026-08-07 07:59:52 on dist-test-slave-zr1t"
I20260807 07:59:52.568948 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:59:52.582568 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:59:52.583967 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:59:52.622525 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.126:46249
I20260807 07:59:52.622623 12705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.126:46249 every 8 connection(s)
I20260807 07:59:52.626377 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:52.635524 12706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4: Bootstrap starting.
I20260807 07:59:52.639825 12706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:52.643865 12706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4: No bootstrap required, opened a new log
I20260807 07:59:52.645838 12706 raft_consensus.cc:348] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e484dc6c578f4ea1acf844cb0e662be4" member_type: VOTER }
I20260807 07:59:52.646317 12706 raft_consensus.cc:374] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:52.646560 12706 raft_consensus.cc:729] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e484dc6c578f4ea1acf844cb0e662be4, State: Initialized, Role: FOLLOWER
I20260807 07:59:52.647181 12706 consensus_queue.cc:260] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e484dc6c578f4ea1acf844cb0e662be4" member_type: VOTER }
I20260807 07:59:52.647642 12706 raft_consensus.cc:388] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 07:59:52.647895 12706 raft_consensus.cc:482] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 07:59:52.648178 12706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:52.651870 12706 raft_consensus.cc:504] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e484dc6c578f4ea1acf844cb0e662be4" member_type: VOTER }
I20260807 07:59:52.652375 12706 leader_election.cc:302] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e484dc6c578f4ea1acf844cb0e662be4; no voters:
I20260807 07:59:52.653463 12706 leader_election.cc:288] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 07:59:52.653970 12710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:52.655309 12710 raft_consensus.cc:686] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 1 LEADER]: Becoming Leader. State: Replica: e484dc6c578f4ea1acf844cb0e662be4, State: Running, Role: LEADER
I20260807 07:59:52.655999 12710 consensus_queue.cc:237] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e484dc6c578f4ea1acf844cb0e662be4" member_type: VOTER }
I20260807 07:59:52.656446 12706 sys_catalog.cc:564] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 07:59:52.661381 12711 sys_catalog.cc:455] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e484dc6c578f4ea1acf844cb0e662be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e484dc6c578f4ea1acf844cb0e662be4" member_type: VOTER } }
I20260807 07:59:52.662053 12711 sys_catalog.cc:458] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [sys.catalog]: This master's current role is: LEADER
I20260807 07:59:52.662465 12712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e484dc6c578f4ea1acf844cb0e662be4. Latest consensus state: current_term: 1 leader_uuid: "e484dc6c578f4ea1acf844cb0e662be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e484dc6c578f4ea1acf844cb0e662be4" member_type: VOTER } }
I20260807 07:59:52.663180 12712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [sys.catalog]: This master's current role is: LEADER
I20260807 07:59:52.663978 12720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 07:59:52.670024 12720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 07:59:52.671595 11825 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260807 07:59:52.678764 12720 catalog_manager.cc:1347] Generated new cluster ID: ca751190f71e41f5b819cf37573ae0d0
I20260807 07:59:52.679003 12720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 07:59:52.693770 12720 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 07:59:52.695256 12720 catalog_manager.cc:1504] Loading token signing keys...
I20260807 07:59:52.718880 12720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4: Generated new TSK 0
I20260807 07:59:52.719519 12720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 07:59:52.737783 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:59:52.743676 12737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:52.744704 12738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:52.746536 12740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:52.747480 11825 server_base.cc:1034] running on GCE node
I20260807 07:59:52.748432 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 07:59:52.748648 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 07:59:52.748821 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089592748807 us; error 0 us; skew 500 ppm
I20260807 07:59:52.749344 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:59:52.751550 11825 webserver.cc:460] Webserver started at http://127.11.140.65:39801/ using document root <none> and password file <none>
I20260807 07:59:52.752043 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 07:59:52.752243 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:59:52.752507 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:59:52.753631 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-0-root/instance:
uuid: "1100218421414a9fa7abd553db3cfca7"
format_stamp: "Formatted at 2026-08-07 07:59:52 on dist-test-slave-zr1t"
I20260807 07:59:52.758008 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260807 07:59:52.761162 12751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:52.761965 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 07:59:52.762197 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-0-root
uuid: "1100218421414a9fa7abd553db3cfca7"
format_stamp: "Formatted at 2026-08-07 07:59:52 on dist-test-slave-zr1t"
I20260807 07:59:52.762452 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:59:52.771845 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:59:52.772943 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:59:52.774336 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:59:52.776540 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:59:52.776703 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:52.776994 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:59:52.777158 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:52.823800 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.65:33141
I20260807 07:59:52.823992 12840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.65:33141 every 8 connection(s)
I20260807 07:59:52.828783 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:59:52.840590 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:52.841850 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:52.847524 12851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:52.847894 11825 server_base.cc:1034] running on GCE node
I20260807 07:59:52.850063 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 07:59:52.850425 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 07:59:52.850713 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089592850691 us; error 0 us; skew 500 ppm
I20260807 07:59:52.851626 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:59:52.855475 11825 webserver.cc:460] Webserver started at http://127.11.140.66:43771/ using document root <none> and password file <none>
I20260807 07:59:52.856317 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 07:59:52.856415 12842 heartbeater.cc:347] Connected to a master server at 127.11.140.126:46249
I20260807 07:59:52.856729 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:59:52.856845 12842 heartbeater.cc:464] Registering TS with master...
I20260807 07:59:52.857187 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:59:52.857708 12842 heartbeater.cc:511] Master 127.11.140.126:46249 requested a full tablet report, sending...
I20260807 07:59:52.859012 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-1-root/instance:
uuid: "166256f885304c169d14729ac6311f16"
format_stamp: "Formatted at 2026-08-07 07:59:52 on dist-test-slave-zr1t"
I20260807 07:59:52.860061 12660 ts_manager.cc:194] Registered new tserver with Master: 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141)
I20260807 07:59:52.863137 12660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51644
I20260807 07:59:52.865931 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 07:59:52.870227 12856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:52.871245 11825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 07:59:52.871565 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-1-root
uuid: "166256f885304c169d14729ac6311f16"
format_stamp: "Formatted at 2026-08-07 07:59:52 on dist-test-slave-zr1t"
I20260807 07:59:52.871871 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:59:52.885862 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:59:52.887117 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:59:52.888764 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:59:52.891269 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:59:52.891508 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:52.891774 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:59:52.891968 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:52.942480 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.66:45069
I20260807 07:59:52.942687 12957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.66:45069 every 8 connection(s)
I20260807 07:59:52.952499 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 07:59:52.960903 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:52.963016 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 07:59:52.965602 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 07:59:52.966166 11825 server_base.cc:1034] running on GCE node
I20260807 07:59:52.967197 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 07:59:52.967447 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 07:59:52.967655 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089592967639 us; error 0 us; skew 500 ppm
I20260807 07:59:52.968408 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 07:59:52.969328 12960 heartbeater.cc:347] Connected to a master server at 127.11.140.126:46249
I20260807 07:59:52.969739 12960 heartbeater.cc:464] Registering TS with master...
I20260807 07:59:52.970516 12960 heartbeater.cc:511] Master 127.11.140.126:46249 requested a full tablet report, sending...
I20260807 07:59:52.971611 11825 webserver.cc:460] Webserver started at http://127.11.140.67:38971/ using document root <none> and password file <none>
I20260807 07:59:52.972165 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 07:59:52.972400 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 07:59:52.972298 12660 ts_manager.cc:194] Registered new tserver with Master: 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:52.972885 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 07:59:52.974169 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-2-root/instance:
uuid: "aaf602cc636547d59d67849689e3c1b7"
format_stamp: "Formatted at 2026-08-07 07:59:52 on dist-test-slave-zr1t"
I20260807 07:59:52.974303 12660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51658
I20260807 07:59:52.978907 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 07:59:52.982365 12971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:52.983116 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 07:59:52.983428 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-2-root
uuid: "aaf602cc636547d59d67849689e3c1b7"
format_stamp: "Formatted at 2026-08-07 07:59:52 on dist-test-slave-zr1t"
I20260807 07:59:52.983712 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786089561396890-11825-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 07:59:52.997494 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 07:59:52.998771 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 07:59:53.000211 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 07:59:53.002547 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 07:59:53.002784 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:53.003053 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 07:59:53.003234 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 07:59:53.045284 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.67:46211
I20260807 07:59:53.045414 13068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.67:46211 every 8 connection(s)
I20260807 07:59:53.063244 13069 heartbeater.cc:347] Connected to a master server at 127.11.140.126:46249
I20260807 07:59:53.063807 13069 heartbeater.cc:464] Registering TS with master...
I20260807 07:59:53.064517 13069 heartbeater.cc:511] Master 127.11.140.126:46249 requested a full tablet report, sending...
I20260807 07:59:53.066620 12660 ts_manager.cc:194] Registered new tserver with Master: aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.067672 11825 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017171848s
I20260807 07:59:53.068267 12660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51670
I20260807 07:59:53.069975 11825 test_util.cc:268] Using random seed: 1673224575
I20260807 07:59:53.095749 12660 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51680:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 07:59:53.099026 12660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 07:59:53.172438 12903 tablet_service.cc:1467] Processing CreateTablet for tablet e186ea2e51aa4dc6b851ce39ba4a37da (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260807 07:59:53.174443 12902 tablet_service.cc:1467] Processing CreateTablet for tablet fd1b31d21185422792582600382ca871 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260807 07:59:53.174621 12904 tablet_service.cc:1467] Processing CreateTablet for tablet 526cbf5f066347a885b5b2ef31f681d1 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260807 07:59:53.175756 12902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1b31d21185422792582600382ca871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.176364 12903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e186ea2e51aa4dc6b851ce39ba4a37da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.177749 12904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526cbf5f066347a885b5b2ef31f681d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.177925 12901 tablet_service.cc:1467] Processing CreateTablet for tablet 73cf365291be4223bf9744b896d0743d (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260807 07:59:53.179227 12901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cf365291be4223bf9744b896d0743d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.179356 12896 tablet_service.cc:1467] Processing CreateTablet for tablet 7392a5e0939b4d7d92f3ead5b9f05a55 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260807 07:59:53.180598 12896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7392a5e0939b4d7d92f3ead5b9f05a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.182143 13010 tablet_service.cc:1467] Processing CreateTablet for tablet 526cbf5f066347a885b5b2ef31f681d1 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260807 07:59:53.182804 12898 tablet_service.cc:1467] Processing CreateTablet for tablet eca6eb08998348c88b1e71c8d0dc71e7 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260807 07:59:53.183416 13010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526cbf5f066347a885b5b2ef31f681d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.184048 12898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca6eb08998348c88b1e71c8d0dc71e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.185920 12899 tablet_service.cc:1467] Processing CreateTablet for tablet a068e349885444c89055c339659e95a5 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260807 07:59:53.186702 13009 tablet_service.cc:1467] Processing CreateTablet for tablet e186ea2e51aa4dc6b851ce39ba4a37da (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260807 07:59:53.187167 12899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a068e349885444c89055c339659e95a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.187981 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e186ea2e51aa4dc6b851ce39ba4a37da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.190682 13008 tablet_service.cc:1467] Processing CreateTablet for tablet fd1b31d21185422792582600382ca871 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260807 07:59:53.191929 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1b31d21185422792582600382ca871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.194708 12894 tablet_service.cc:1467] Processing CreateTablet for tablet 06b99890fa394a81853cfc4805b31172 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260807 07:59:53.195222 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 73cf365291be4223bf9744b896d0743d (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260807 07:59:53.195983 12894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b99890fa394a81853cfc4805b31172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.196523 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cf365291be4223bf9744b896d0743d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.198724 13003 tablet_service.cc:1467] Processing CreateTablet for tablet eca6eb08998348c88b1e71c8d0dc71e7 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260807 07:59:53.199198 12999 tablet_service.cc:1467] Processing CreateTablet for tablet a532ddc55e4e421a9ab42b09bb326b41 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260807 07:59:53.200018 13003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca6eb08998348c88b1e71c8d0dc71e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.201447 13000 tablet_service.cc:1467] Processing CreateTablet for tablet 06b99890fa394a81853cfc4805b31172 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260807 07:59:53.202806 13000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b99890fa394a81853cfc4805b31172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.202716 12998 tablet_service.cc:1467] Processing CreateTablet for tablet 933df2ca62e24d5da058bb135d66eac4 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260807 07:59:53.204005 12998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933df2ca62e24d5da058bb135d66eac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.206768 13001 tablet_service.cc:1467] Processing CreateTablet for tablet 7392a5e0939b4d7d92f3ead5b9f05a55 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260807 07:59:53.206786 12892 tablet_service.cc:1467] Processing CreateTablet for tablet a532ddc55e4e421a9ab42b09bb326b41 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260807 07:59:53.208189 13001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7392a5e0939b4d7d92f3ead5b9f05a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.208613 12892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a532ddc55e4e421a9ab42b09bb326b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.209895 12999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a532ddc55e4e421a9ab42b09bb326b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.220410 12889 tablet_service.cc:1467] Processing CreateTablet for tablet 933df2ca62e24d5da058bb135d66eac4 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260807 07:59:53.224532 13004 tablet_service.cc:1467] Processing CreateTablet for tablet a068e349885444c89055c339659e95a5 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260807 07:59:53.225800 13004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a068e349885444c89055c339659e95a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.230917 12889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933df2ca62e24d5da058bb135d66eac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.297986 12790 tablet_service.cc:1467] Processing CreateTablet for tablet 526cbf5f066347a885b5b2ef31f681d1 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260807 07:59:53.299345 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526cbf5f066347a885b5b2ef31f681d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.302790 12787 tablet_service.cc:1467] Processing CreateTablet for tablet e186ea2e51aa4dc6b851ce39ba4a37da (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260807 07:59:53.304138 12787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e186ea2e51aa4dc6b851ce39ba4a37da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.306411 12783 tablet_service.cc:1467] Processing CreateTablet for tablet eca6eb08998348c88b1e71c8d0dc71e7 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260807 07:59:53.306684 12781 tablet_service.cc:1467] Processing CreateTablet for tablet 06b99890fa394a81853cfc4805b31172 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260807 07:59:53.307793 12783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca6eb08998348c88b1e71c8d0dc71e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.308070 12786 tablet_service.cc:1467] Processing CreateTablet for tablet fd1b31d21185422792582600382ca871 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260807 07:59:53.310636 12785 tablet_service.cc:1467] Processing CreateTablet for tablet 73cf365291be4223bf9744b896d0743d (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260807 07:59:53.311726 12780 tablet_service.cc:1467] Processing CreateTablet for tablet a532ddc55e4e421a9ab42b09bb326b41 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260807 07:59:53.312577 12784 tablet_service.cc:1467] Processing CreateTablet for tablet a068e349885444c89055c339659e95a5 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260807 07:59:53.310644 12782 tablet_service.cc:1467] Processing CreateTablet for tablet 7392a5e0939b4d7d92f3ead5b9f05a55 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260807 07:59:53.314292 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1b31d21185422792582600382ca871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.314805 12779 tablet_service.cc:1467] Processing CreateTablet for tablet 933df2ca62e24d5da058bb135d66eac4 (DEFAULT_TABLE table=test-workload [id=711c5eb650e84987b9f6a81d42698333]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260807 07:59:53.316814 12782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7392a5e0939b4d7d92f3ead5b9f05a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.317036 13088 tablet_bootstrap.cc:492] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:53.318462 12781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b99890fa394a81853cfc4805b31172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.322079 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a068e349885444c89055c339659e95a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.322746 12780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a532ddc55e4e421a9ab42b09bb326b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.324483 13088 tablet_bootstrap.cc:654] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.326006 12779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933df2ca62e24d5da058bb135d66eac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.329895 12785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cf365291be4223bf9744b896d0743d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 07:59:53.346228 13086 tablet_bootstrap.cc:492] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:53.385577 13086 tablet_bootstrap.cc:654] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.422610 13090 tablet_bootstrap.cc:492] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:53.428375 13090 tablet_bootstrap.cc:654] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.428843 13088 tablet_bootstrap.cc:492] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:53.429395 13088 ts_tablet_manager.cc:1397] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.113s user 0.040s sys 0.066s
I20260807 07:59:53.430842 13086 tablet_bootstrap.cc:492] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:53.431412 13086 ts_tablet_manager.cc:1397] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.009s
I20260807 07:59:53.432276 13088 raft_consensus.cc:348] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.433199 13088 raft_consensus.cc:374] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.433578 13088 raft_consensus.cc:729] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.434208 13086 raft_consensus.cc:348] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.434291 13088 consensus_queue.cc:260] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.434868 13090 tablet_bootstrap.cc:492] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:53.434825 13086 raft_consensus.cc:374] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.435320 13086 raft_consensus.cc:729] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.435395 13090 ts_tablet_manager.cc:1397] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260807 07:59:53.436656 13069 heartbeater.cc:503] Master 127.11.140.126:46249 was elected leader, sending a full tablet report...
I20260807 07:59:53.436246 13086 consensus_queue.cc:260] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.437111 13088 ts_tablet_manager.cc:1428] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260807 07:59:53.438024 13088 tablet_bootstrap.cc:492] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:53.438295 13090 raft_consensus.cc:348] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:53.439035 13090 raft_consensus.cc:374] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.439352 13090 raft_consensus.cc:729] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.440058 13090 consensus_queue.cc:260] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:53.445219 13088 tablet_bootstrap.cc:654] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.469599 12842 heartbeater.cc:503] Master 127.11.140.126:46249 was elected leader, sending a full tablet report...
I20260807 07:59:53.469740 12960 heartbeater.cc:503] Master 127.11.140.126:46249 was elected leader, sending a full tablet report...
I20260807 07:59:53.470395 13086 ts_tablet_manager.cc:1428] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.039s user 0.026s sys 0.010s
I20260807 07:59:53.471480 13086 tablet_bootstrap.cc:492] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:53.474047 13090 ts_tablet_manager.cc:1428] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.038s user 0.014s sys 0.019s
I20260807 07:59:53.475107 13090 tablet_bootstrap.cc:492] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:53.477559 13086 tablet_bootstrap.cc:654] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.479321 13088 tablet_bootstrap.cc:492] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:53.479974 13088 ts_tablet_manager.cc:1397] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.014s
I20260807 07:59:53.480990 13090 tablet_bootstrap.cc:654] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.486807 13088 raft_consensus.cc:348] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:53.487584 13088 raft_consensus.cc:374] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.487888 13088 raft_consensus.cc:729] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.488469 13086 tablet_bootstrap.cc:492] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:53.489137 13086 ts_tablet_manager.cc:1397] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.006s
I20260807 07:59:53.488664 13088 consensus_queue.cc:260] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:53.491354 13088 ts_tablet_manager.cc:1428] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260807 07:59:53.492220 13088 tablet_bootstrap.cc:492] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:53.492660 13086 raft_consensus.cc:348] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.493762 13086 raft_consensus.cc:374] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.494197 13086 raft_consensus.cc:729] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.495338 13086 consensus_queue.cc:260] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.498210 13086 ts_tablet_manager.cc:1428] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260807 07:59:53.498660 13088 tablet_bootstrap.cc:654] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.499354 13086 tablet_bootstrap.cc:492] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:53.501358 13090 tablet_bootstrap.cc:492] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:53.501899 13090 ts_tablet_manager.cc:1397] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.017s
I20260807 07:59:53.504613 13090 raft_consensus.cc:348] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.505398 13090 raft_consensus.cc:374] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.505777 13090 raft_consensus.cc:729] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.505934 13086 tablet_bootstrap.cc:654] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.506711 13090 consensus_queue.cc:260] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.509421 13090 ts_tablet_manager.cc:1428] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260807 07:59:53.510593 13090 tablet_bootstrap.cc:492] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:53.517505 13086 tablet_bootstrap.cc:492] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:53.517840 13090 tablet_bootstrap.cc:654] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.518091 13086 ts_tablet_manager.cc:1397] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20260807 07:59:53.521070 13086 raft_consensus.cc:348] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.522316 13086 raft_consensus.cc:374] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.522763 13086 raft_consensus.cc:729] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.523846 13086 consensus_queue.cc:260] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.526723 13086 ts_tablet_manager.cc:1428] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260807 07:59:53.527801 13086 tablet_bootstrap.cc:492] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:53.529621 13090 tablet_bootstrap.cc:492] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:53.530143 13090 ts_tablet_manager.cc:1397] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20260807 07:59:53.532809 13090 raft_consensus.cc:348] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.533713 13086 tablet_bootstrap.cc:654] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.533713 13090 raft_consensus.cc:374] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.534242 13090 raft_consensus.cc:729] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.535077 13090 consensus_queue.cc:260] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.538069 13090 ts_tablet_manager.cc:1428] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260807 07:59:53.539381 13090 tablet_bootstrap.cc:492] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:53.545483 13086 tablet_bootstrap.cc:492] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:53.546082 13086 ts_tablet_manager.cc:1397] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.017s
I20260807 07:59:53.546362 13090 tablet_bootstrap.cc:654] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.548828 13086 raft_consensus.cc:348] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.550021 13086 raft_consensus.cc:374] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.550460 13086 raft_consensus.cc:729] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.551446 13086 consensus_queue.cc:260] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.554491 13086 ts_tablet_manager.cc:1428] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260807 07:59:53.555756 13086 tablet_bootstrap.cc:492] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:53.558861 13090 tablet_bootstrap.cc:492] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:53.559446 13090 ts_tablet_manager.cc:1397] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.003s
I20260807 07:59:53.561918 13086 tablet_bootstrap.cc:654] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.562078 13090 raft_consensus.cc:348] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.563145 13090 raft_consensus.cc:374] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.563506 13090 raft_consensus.cc:729] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.564419 13090 consensus_queue.cc:260] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.569744 13090 ts_tablet_manager.cc:1428] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20260807 07:59:53.571008 13090 tablet_bootstrap.cc:492] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:53.574008 13086 tablet_bootstrap.cc:492] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:53.574587 13086 ts_tablet_manager.cc:1397] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.005s
I20260807 07:59:53.577024 13090 tablet_bootstrap.cc:654] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.577369 13086 raft_consensus.cc:348] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.578265 13086 raft_consensus.cc:374] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.578617 13086 raft_consensus.cc:729] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.579475 13086 consensus_queue.cc:260] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.582820 13086 ts_tablet_manager.cc:1428] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260807 07:59:53.584134 13086 tablet_bootstrap.cc:492] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:53.585774 13088 tablet_bootstrap.cc:492] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:53.586277 13088 ts_tablet_manager.cc:1397] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.094s user 0.040s sys 0.050s
I20260807 07:59:53.589118 13088 raft_consensus.cc:348] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.589969 13088 raft_consensus.cc:374] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.590296 13088 raft_consensus.cc:729] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.591135 13088 consensus_queue.cc:260] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.592058 13086 tablet_bootstrap.cc:654] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.595956 13088 ts_tablet_manager.cc:1428] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260807 07:59:53.596949 13088 tablet_bootstrap.cc:492] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:53.602844 13088 tablet_bootstrap.cc:654] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.604851 13086 tablet_bootstrap.cc:492] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:53.605574 13086 ts_tablet_manager.cc:1397] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.002s
I20260807 07:59:53.609045 13086 raft_consensus.cc:348] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.610366 13086 raft_consensus.cc:374] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.610802 13086 raft_consensus.cc:729] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.613325 13095 raft_consensus.cc:482] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.613977 13088 tablet_bootstrap.cc:492] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:53.614555 13088 ts_tablet_manager.cc:1397] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.011s
I20260807 07:59:53.613931 13095 raft_consensus.cc:504] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.618193 13095 leader_election.cc:288] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:53.617157 13088 raft_consensus.cc:348] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.611941 13086 consensus_queue.cc:260] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.618937 13088 raft_consensus.cc:374] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.619447 13088 raft_consensus.cc:729] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.620527 13088 consensus_queue.cc:260] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.623633 13088 ts_tablet_manager.cc:1428] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260807 07:59:53.624549 13088 tablet_bootstrap.cc:492] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:53.625057 13090 tablet_bootstrap.cc:492] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:53.627310 13090 ts_tablet_manager.cc:1397] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.056s user 0.022s sys 0.024s
I20260807 07:59:53.628885 13086 ts_tablet_manager.cc:1428] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20260807 07:59:53.630648 13088 tablet_bootstrap.cc:654] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.631678 13090 raft_consensus.cc:348] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.633919 13090 raft_consensus.cc:374] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.634132 13097 raft_consensus.cc:482] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.634318 13086 tablet_bootstrap.cc:492] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:53.634310 13090 raft_consensus.cc:729] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.635373 13090 consensus_queue.cc:260] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.634615 13097 raft_consensus.cc:504] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.639163 13097 leader_election.cc:288] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.641286 13086 tablet_bootstrap.cc:654] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.657039 13090 ts_tablet_manager.cc:1428] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.029s user 0.008s sys 0.003s
I20260807 07:59:53.658186 13090 tablet_bootstrap.cc:492] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:53.660228 13096 raft_consensus.cc:482] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.660694 13096 raft_consensus.cc:504] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.663232 13096 leader_election.cc:288] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.665745 13088 tablet_bootstrap.cc:492] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:53.666249 13088 ts_tablet_manager.cc:1397] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.002s
I20260807 07:59:53.666401 13090 tablet_bootstrap.cc:654] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
W20260807 07:59:53.667361 12723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 07:59:53.669201 13088 raft_consensus.cc:348] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.670260 13088 raft_consensus.cc:374] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.670773 13088 raft_consensus.cc:729] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.673511 13096 raft_consensus.cc:482] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.674036 13096 raft_consensus.cc:504] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.671720 13088 consensus_queue.cc:260] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.675860 13096 leader_election.cc:288] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.676255 13088 ts_tablet_manager.cc:1428] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20260807 07:59:53.677654 13088 tablet_bootstrap.cc:492] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:53.684952 13088 tablet_bootstrap.cc:654] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.702062 13095 raft_consensus.cc:482] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.702612 13095 raft_consensus.cc:504] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:53.704737 13095 leader_election.cc:288] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:53.733568 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b31d21185422792582600382ca871" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:53.735746 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:53.736657 12803 raft_consensus.cc:2454] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 0.
W20260807 07:59:53.738653 12977 leader_election.cc:341] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.739636 12977 leader_election.cc:302] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:53.740746 13095 raft_consensus.cc:2781] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:53.741175 13095 raft_consensus.cc:482] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:53.741580 13095 raft_consensus.cc:3037] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:53.751282 13095 raft_consensus.cc:504] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:53.753815 13122 raft_consensus.cc:482] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.754444 13122 raft_consensus.cc:504] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.757720 13122 leader_election.cc:288] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.782317 12922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933df2ca62e24d5da058bb135d66eac4" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:53.784920 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:53.785683 12921 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 0.
W20260807 07:59:53.787364 12752 leader_election.cc:341] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 166256f885304c169d14729ac6311f16 (127.11.140.66:45069): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.788292 12752 leader_election.cc:302] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:53.789505 13122 raft_consensus.cc:2781] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:53.790050 13122 raft_consensus.cc:482] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:53.790520 13122 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:53.794440 13095 leader_election.cc:288] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:53.796535 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933df2ca62e24d5da058bb135d66eac4" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:53.797528 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:53.809653 13028 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 0.
I20260807 07:59:53.802476 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b31d21185422792582600382ca871" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:53.803586 12922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:53.805074 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:53.807283 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:53.814643 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:53.815392 12803 raft_consensus.cc:3037] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
W20260807 07:59:53.822233 12973 leader_election.cc:341] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 166256f885304c169d14729ac6311f16 (127.11.140.66:45069): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.822840 12973 leader_election.cc:302] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aaf602cc636547d59d67849689e3c1b7; no voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16
I20260807 07:59:53.832877 13140 raft_consensus.cc:2726] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:53.843068 13122 raft_consensus.cc:504] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.843547 13086 tablet_bootstrap.cc:492] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:53.843621 12803 raft_consensus.cc:2454] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 1.
I20260807 07:59:53.844326 13086 ts_tablet_manager.cc:1397] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.210s user 0.067s sys 0.117s
I20260807 07:59:53.839018 13096 raft_consensus.cc:482] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.845283 13122 leader_election.cc:288] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 election: Requested vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.845433 13096 raft_consensus.cc:504] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.846747 13086 raft_consensus.cc:348] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.847235 13086 raft_consensus.cc:374] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.847401 13086 raft_consensus.cc:729] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.847960 13096 leader_election.cc:288] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.847939 13086 consensus_queue.cc:260] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.848909 13140 raft_consensus.cc:482] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.851302 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:53.851889 12920 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:53.852037 13086 ts_tablet_manager.cc:1428] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260807 07:59:53.852805 13096 raft_consensus.cc:482] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.854177 13140 raft_consensus.cc:504] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.854882 13096 raft_consensus.cc:504] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.831780 13095 raft_consensus.cc:482] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.860090 13140 leader_election.cc:288] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:53.860046 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
W20260807 07:59:53.860708 12973 leader_election.cc:341] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 166256f885304c169d14729ac6311f16 (127.11.140.66:45069): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.861079 12803 raft_consensus.cc:2379] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 166256f885304c169d14729ac6311f16 in current term 1: Already voted for candidate 1100218421414a9fa7abd553db3cfca7 in this term.
I20260807 07:59:53.856129 12977 leader_election.cc:302] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
W20260807 07:59:53.862501 12864 leader_election.cc:341] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Illegal state: must be running to vote when last-logged opid is not known
W20260807 07:59:53.862969 12973 leader_election.cc:341] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 166256f885304c169d14729ac6311f16 (127.11.140.66:45069): Illegal state: must be running to vote when last-logged opid is not known
W20260807 07:59:53.863713 12756 leader_election.cc:341] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.863844 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:53.866796 13096 leader_election.cc:288] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.870724 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:53.868199 12756 leader_election.cc:302] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7; no voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7
I20260807 07:59:53.872148 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:53.873270 13026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:53.873915 13022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a532ddc55e4e421a9ab42b09bb326b41" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:53.874924 12922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:53.875574 12922 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 0.
I20260807 07:59:53.875166 13021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a068e349885444c89055c339659e95a5" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:53.876570 13088 tablet_bootstrap.cc:492] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:53.877115 13021 raft_consensus.cc:2454] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 0.
I20260807 07:59:53.872735 13029 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 0.
I20260807 07:59:53.872732 13122 raft_consensus.cc:2726] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:53.869205 13090 tablet_bootstrap.cc:492] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:53.878687 13090 ts_tablet_manager.cc:1397] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.221s user 0.047s sys 0.067s
I20260807 07:59:53.880527 12920 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 1.
I20260807 07:59:53.881395 13090 raft_consensus.cc:348] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.882138 13090 raft_consensus.cc:374] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.882479 13090 raft_consensus.cc:729] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
W20260807 07:59:53.882510 12864 leader_election.cc:341] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.883448 12864 leader_election.cc:302] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
W20260807 07:59:53.876504 12977 leader_election.cc:341] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.884758 12973 leader_election.cc:302] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters: 1100218421414a9fa7abd553db3cfca7
I20260807 07:59:53.884956 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a532ddc55e4e421a9ab42b09bb326b41" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:53.885792 12803 raft_consensus.cc:2454] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 0.
I20260807 07:59:53.886286 13140 raft_consensus.cc:2781] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:53.886384 12752 leader_election.cc:302] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:53.886727 13140 raft_consensus.cc:482] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:53.887109 13140 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:53.887233 13122 raft_consensus.cc:2781] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:53.887332 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a068e349885444c89055c339659e95a5" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:53.887756 13122 raft_consensus.cc:686] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 1 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 07:59:53.888602 13122 consensus_queue.cc:237] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.883250 13090 consensus_queue.cc:260] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.864522 13139 raft_consensus.cc:2781] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:53.859568 13095 raft_consensus.cc:504] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.896819 13095 leader_election.cc:288] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:53.869148 13086 tablet_bootstrap.cc:492] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:53.897549 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:53.900799 13028 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:53.901662 13139 raft_consensus.cc:686] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Becoming Leader. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 07:59:53.902096 13086 tablet_bootstrap.cc:654] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.901880 12659 catalog_manager.cc:5495] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 1 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:53.902573 13139 consensus_queue.cc:237] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:53.894447 13140 raft_consensus.cc:504] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.908061 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:53.909250 13028 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 1.
I20260807 07:59:53.886580 12864 leader_election.cc:302] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters: 1100218421414a9fa7abd553db3cfca7
I20260807 07:59:53.894474 13142 raft_consensus.cc:2781] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:53.910347 13142 raft_consensus.cc:482] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:53.910833 13142 raft_consensus.cc:3037] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:53.877853 13088 ts_tablet_manager.cc:1397] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.200s user 0.061s sys 0.103s
W20260807 07:59:53.909385 12977 leader_election.cc:341] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.916774 13140 leader_election.cc:288] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:53.916658 13088 raft_consensus.cc:348] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.917676 13088 raft_consensus.cc:374] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.918063 13088 raft_consensus.cc:729] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.919013 13088 consensus_queue.cc:260] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.921864 13088 ts_tablet_manager.cc:1428] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260807 07:59:53.922853 13088 tablet_bootstrap.cc:492] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:53.923162 13142 raft_consensus.cc:504] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.923895 13086 tablet_bootstrap.cc:492] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:53.924348 13086 ts_tablet_manager.cc:1397] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.014s
I20260807 07:59:53.925113 13142 leader_election.cc:288] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.927203 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a068e349885444c89055c339659e95a5" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:53.926980 13086 raft_consensus.cc:348] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.927726 12660 catalog_manager.cc:5495] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf602cc636547d59d67849689e3c1b7 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "aaf602cc636547d59d67849689e3c1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:53.928819 13088 tablet_bootstrap.cc:654] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.928790 13086 raft_consensus.cc:374] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.928838 13090 ts_tablet_manager.cc:1428] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20260807 07:59:53.929241 13086 raft_consensus.cc:729] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.930619 13090 tablet_bootstrap.cc:492] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:53.931120 13096 raft_consensus.cc:2781] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:53.930856 13086 consensus_queue.cc:260] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.931506 13096 raft_consensus.cc:482] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:53.931867 13096 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:53.936894 13090 tablet_bootstrap.cc:654] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.937899 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:53.938740 12920 raft_consensus.cc:2379] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aaf602cc636547d59d67849689e3c1b7 in current term 1: Already voted for candidate 1100218421414a9fa7abd553db3cfca7 in this term.
I20260807 07:59:53.938072 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a068e349885444c89055c339659e95a5" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:53.940001 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:53.938941 13096 raft_consensus.cc:504] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.939967 13028 raft_consensus.cc:3037] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:53.941129 12920 raft_consensus.cc:2379] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aaf602cc636547d59d67849689e3c1b7 in current term 1: Already voted for candidate 166256f885304c169d14729ac6311f16 in this term.
I20260807 07:59:53.942768 13096 leader_election.cc:288] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.943243 13096 raft_consensus.cc:482] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.943647 13096 raft_consensus.cc:504] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.945426 13096 leader_election.cc:288] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.945868 12864 leader_election.cc:302] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 166256f885304c169d14729ac6311f16; no voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7
I20260807 07:59:53.946954 12864 leader_election.cc:302] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters: aaf602cc636547d59d67849689e3c1b7
W20260807 07:59:53.948014 12864 leader_election.cc:341] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.948868 13096 raft_consensus.cc:2726] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:53.949345 13142 raft_consensus.cc:2781] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:53.949793 13142 raft_consensus.cc:482] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:53.950171 13142 raft_consensus.cc:3037] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
W20260807 07:59:53.948835 12864 leader_election.cc:341] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.958421 12973 leader_election.cc:302] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aaf602cc636547d59d67849689e3c1b7; no voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16
I20260807 07:59:53.959671 13139 raft_consensus.cc:2726] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:53.960500 13086 ts_tablet_manager.cc:1428] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.036s user 0.016s sys 0.017s
I20260807 07:59:53.960464 13154 raft_consensus.cc:482] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:53.960865 12973 leader_election.cc:302] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aaf602cc636547d59d67849689e3c1b7; no voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16
I20260807 07:59:53.961108 13154 raft_consensus.cc:504] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.961992 13139 raft_consensus.cc:2726] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 07:59:53.963733 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:53.964044 13088 tablet_bootstrap.cc:492] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:53.964205 13086 tablet_bootstrap.cc:492] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:53.964485 13088 ts_tablet_manager.cc:1397] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.011s
I20260807 07:59:53.966806 13154 leader_election.cc:288] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.966857 13028 raft_consensus.cc:2454] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 1.
I20260807 07:59:53.968886 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:53.969381 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:53.969692 13029 raft_consensus.cc:2379] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 166256f885304c169d14729ac6311f16 in current term 1: Already voted for candidate aaf602cc636547d59d67849689e3c1b7 in this term.
I20260807 07:59:53.971279 13086 tablet_bootstrap.cc:654] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:53.971410 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:53.972159 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:53.972771 12803 raft_consensus.cc:2454] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 0.
I20260807 07:59:53.972731 13088 raft_consensus.cc:348] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.973644 13088 raft_consensus.cc:374] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.973944 13088 raft_consensus.cc:729] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.974629 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
W20260807 07:59:53.974629 12864 leader_election.cc:341] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Illegal state: must be running to vote when last-logged opid is not known
W20260807 07:59:53.976413 12864 leader_election.cc:341] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.977119 12864 leader_election.cc:302] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:53.978137 13088 consensus_queue.cc:260] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.978343 13142 raft_consensus.cc:504] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.980499 13088 ts_tablet_manager.cc:1428] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.016s user 0.003s sys 0.003s
I20260807 07:59:53.980672 13142 leader_election.cc:288] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:53.981455 13088 tablet_bootstrap.cc:492] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:53.983457 13090 tablet_bootstrap.cc:492] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:53.983980 13090 ts_tablet_manager.cc:1397] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.004s
I20260807 07:59:53.984138 13154 raft_consensus.cc:2781] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:53.984488 13154 raft_consensus.cc:482] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:53.984830 13154 raft_consensus.cc:3037] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:53.986961 13090 raft_consensus.cc:348] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.987707 13090 raft_consensus.cc:374] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:53.988010 13090 raft_consensus.cc:729] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
I20260807 07:59:53.988881 13090 consensus_queue.cc:260] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.991393 13090 ts_tablet_manager.cc:1428] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260807 07:59:53.992290 13090 tablet_bootstrap.cc:492] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:53.992172 13154 raft_consensus.cc:504] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.993852 12864 leader_election.cc:302] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters: 1100218421414a9fa7abd553db3cfca7
I20260807 07:59:53.994591 13142 raft_consensus.cc:2781] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:53.995033 12864 leader_election.cc:302] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 166256f885304c169d14729ac6311f16; no voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7
I20260807 07:59:53.995042 13142 raft_consensus.cc:686] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Becoming Leader. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 07:59:53.996058 13142 consensus_queue.cc:237] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:53.998251 13096 raft_consensus.cc:2726] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260807 07:59:53.999286 12864 leader_election.cc:341] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:53.999838 12864 leader_election.cc:302] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 166256f885304c169d14729ac6311f16; no voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7
I20260807 07:59:54.000612 13096 raft_consensus.cc:2726] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 07:59:54.001726 12864 leader_election.cc:341] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:54.003430 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a532ddc55e4e421a9ab42b09bb326b41" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:54.004041 12803 raft_consensus.cc:3037] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.006791 13154 leader_election.cc:288] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:54.008352 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a532ddc55e4e421a9ab42b09bb326b41" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:54.009112 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:54.009688 12804 raft_consensus.cc:3037] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.012095 12803 raft_consensus.cc:2454] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 1.
I20260807 07:59:54.016983 12804 raft_consensus.cc:2454] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 1.
I20260807 07:59:54.017407 13090 tablet_bootstrap.cc:654] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:54.021346 12864 leader_election.cc:302] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:54.022708 12864 leader_election.cc:302] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:54.023587 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:54.025081 13088 tablet_bootstrap.cc:654] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:54.025684 13086 tablet_bootstrap.cc:492] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:54.025655 13096 raft_consensus.cc:2781] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:54.026234 13086 ts_tablet_manager.cc:1397] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.062s user 0.037s sys 0.023s
I20260807 07:59:54.026698 12660 catalog_manager.cc:5495] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 166256f885304c169d14729ac6311f16 (127.11.140.66). New cstate: current_term: 1 leader_uuid: "166256f885304c169d14729ac6311f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:54.028853 13142 raft_consensus.cc:2781] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:54.028671 13086 raft_consensus.cc:348] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:54.029206 13154 raft_consensus.cc:482] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:54.029351 13142 raft_consensus.cc:686] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Becoming Leader. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 07:59:54.029394 13086 raft_consensus.cc:374] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:54.029850 13086 raft_consensus.cc:729] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:54.029695 13154 raft_consensus.cc:504] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.030267 13142 consensus_queue.cc:237] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.030505 13086 consensus_queue.cc:260] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:54.031592 13154 leader_election.cc:288] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
W20260807 07:59:54.035197 12864 leader_election.cc:341] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211): Illegal state: must be running to vote when last-logged opid is not known
W20260807 07:59:54.036111 12864 leader_election.cc:341] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211): Illegal state: must be running to vote when last-logged opid is not known
I20260807 07:59:54.037071 13090 tablet_bootstrap.cc:492] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:54.037503 13090 ts_tablet_manager.cc:1397] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.000s
I20260807 07:59:54.037761 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:54.038420 12804 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 0.
I20260807 07:59:54.039410 13086 ts_tablet_manager.cc:1428] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260807 07:59:54.040194 13086 tablet_bootstrap.cc:492] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16: Bootstrap starting.
I20260807 07:59:54.040076 13090 raft_consensus.cc:348] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.040781 13090 raft_consensus.cc:374] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:54.041078 13090 raft_consensus.cc:729] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
I20260807 07:59:54.044780 13086 tablet_bootstrap.cc:654] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:54.046417 13090 consensus_queue.cc:260] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.047677 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:54.048282 13028 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 0.
I20260807 07:59:54.048522 13090 ts_tablet_manager.cc:1428] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260807 07:59:54.049412 12864 leader_election.cc:302] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:54.051227 13142 raft_consensus.cc:2781] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:54.051564 13142 raft_consensus.cc:482] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:54.051903 13142 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.052510 13090 tablet_bootstrap.cc:492] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:54.055449 13086 tablet_bootstrap.cc:492] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16: No bootstrap required, opened a new log
I20260807 07:59:54.055908 13086 ts_tablet_manager.cc:1397] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260807 07:59:54.058454 13090 tablet_bootstrap.cc:654] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:54.058496 13086 raft_consensus.cc:348] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.059293 13086 raft_consensus.cc:374] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:54.059571 13086 raft_consensus.cc:729] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Initialized, Role: FOLLOWER
I20260807 07:59:54.060300 13086 consensus_queue.cc:260] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.061122 13142 raft_consensus.cc:504] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.062901 13086 ts_tablet_manager.cc:1428] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260807 07:59:54.063572 13088 tablet_bootstrap.cc:492] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:54.063534 13142 leader_election.cc:288] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:54.064160 13088 ts_tablet_manager.cc:1397] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.083s user 0.010s sys 0.002s
I20260807 07:59:54.064211 13096 raft_consensus.cc:686] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Becoming Leader. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 07:59:54.065655 13090 tablet_bootstrap.cc:492] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:54.065157 13096 consensus_queue.cc:237] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.066118 13090 ts_tablet_manager.cc:1397] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20260807 07:59:54.066897 13088 raft_consensus.cc:348] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.067584 13088 raft_consensus.cc:374] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:54.067454 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:54.067883 13088 raft_consensus.cc:729] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:54.068058 12804 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.068485 13090 raft_consensus.cc:348] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.069205 13090 raft_consensus.cc:374] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:54.068691 13088 consensus_queue.cc:260] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.069502 13090 raft_consensus.cc:729] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
I20260807 07:59:54.070125 13090 consensus_queue.cc:260] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.071846 13088 ts_tablet_manager.cc:1428] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 07:59:54.072197 13090 ts_tablet_manager.cc:1428] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260807 07:59:54.072870 13088 tablet_bootstrap.cc:492] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:54.073035 13090 tablet_bootstrap.cc:492] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7: Bootstrap starting.
I20260807 07:59:54.079823 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:54.080371 13028 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.084260 13090 tablet_bootstrap.cc:654] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:54.086771 12804 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 1.
I20260807 07:59:54.086625 12656 catalog_manager.cc:5495] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 166256f885304c169d14729ac6311f16 (127.11.140.66). New cstate: current_term: 1 leader_uuid: "166256f885304c169d14729ac6311f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:54.089044 12656 catalog_manager.cc:5495] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 166256f885304c169d14729ac6311f16 (127.11.140.66). New cstate: current_term: 1 leader_uuid: "166256f885304c169d14729ac6311f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:54.091192 13028 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 1.
I20260807 07:59:54.093317 13088 tablet_bootstrap.cc:654] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:54.097612 13088 tablet_bootstrap.cc:492] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:54.097709 13122 raft_consensus.cc:482] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:54.098151 13122 raft_consensus.cc:504] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.099030 12864 leader_election.cc:302] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:54.100104 13122 leader_election.cc:288] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:54.100633 13096 raft_consensus.cc:482] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:54.101747 13142 raft_consensus.cc:2781] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:54.102240 13142 raft_consensus.cc:686] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Becoming Leader. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 07:59:54.102557 13090 tablet_bootstrap.cc:492] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7: No bootstrap required, opened a new log
I20260807 07:59:54.102946 13090 ts_tablet_manager.cc:1397] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.001s
I20260807 07:59:54.103085 13142 consensus_queue.cc:237] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.105969 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:54.106226 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:54.106727 12920 raft_consensus.cc:2379] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1100218421414a9fa7abd553db3cfca7 in current term 1: Already voted for candidate 166256f885304c169d14729ac6311f16 in this term.
I20260807 07:59:54.106943 13028 raft_consensus.cc:2379] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1100218421414a9fa7abd553db3cfca7 in current term 1: Already voted for candidate aaf602cc636547d59d67849689e3c1b7 in this term.
I20260807 07:59:54.108083 12752 leader_election.cc:302] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7; no voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7
I20260807 07:59:54.098151 13088 ts_tablet_manager.cc:1397] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.001s
I20260807 07:59:54.105270 13090 raft_consensus.cc:348] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.101109 13096 raft_consensus.cc:504] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:54.109362 13122 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.111043 13096 leader_election.cc:288] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211), 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141)
I20260807 07:59:54.111517 13090 raft_consensus.cc:374] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:54.111788 13090 raft_consensus.cc:729] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Initialized, Role: FOLLOWER
I20260807 07:59:54.112000 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:54.112004 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:54.112370 13090 consensus_queue.cc:260] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.112862 12804 raft_consensus.cc:2379] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 166256f885304c169d14729ac6311f16 in current term 1: Already voted for candidate aaf602cc636547d59d67849689e3c1b7 in this term.
I20260807 07:59:54.114053 12864 leader_election.cc:302] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 166256f885304c169d14729ac6311f16; no voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7
I20260807 07:59:54.114773 13096 raft_consensus.cc:3037] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.116022 13088 raft_consensus.cc:348] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.116693 13088 raft_consensus.cc:374] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:54.116925 13088 raft_consensus.cc:729] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:54.118083 13090 ts_tablet_manager.cc:1428] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260807 07:59:54.118882 13088 consensus_queue.cc:260] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.120486 13122 raft_consensus.cc:2726] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:54.121212 13088 ts_tablet_manager.cc:1428] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260807 07:59:54.120731 12660 catalog_manager.cc:5495] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 166256f885304c169d14729ac6311f16 (127.11.140.66). New cstate: current_term: 1 leader_uuid: "166256f885304c169d14729ac6311f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:54.121999 13088 tablet_bootstrap.cc:492] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7: Bootstrap starting.
I20260807 07:59:54.122875 13096 raft_consensus.cc:2726] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:54.127417 13088 tablet_bootstrap.cc:654] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7: Neither blocks nor log segments found. Creating new log.
I20260807 07:59:54.131943 13088 tablet_bootstrap.cc:492] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7: No bootstrap required, opened a new log
I20260807 07:59:54.132289 13088 ts_tablet_manager.cc:1397] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260807 07:59:54.134487 13088 raft_consensus.cc:348] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.135157 13088 raft_consensus.cc:374] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 07:59:54.135416 13088 raft_consensus.cc:729] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Initialized, Role: FOLLOWER
I20260807 07:59:54.135610 13122 raft_consensus.cc:482] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:54.136065 13088 consensus_queue.cc:260] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.136092 13122 raft_consensus.cc:504] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.138087 13088 ts_tablet_manager.cc:1428] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 07:59:54.138796 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b31d21185422792582600382ca871" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:54.139072 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b31d21185422792582600382ca871" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7" is_pre_election: true
I20260807 07:59:54.139458 12920 raft_consensus.cc:2454] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 0.
I20260807 07:59:54.139658 13028 raft_consensus.cc:2454] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 0.
I20260807 07:59:54.140376 12752 leader_election.cc:302] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:54.140640 13122 leader_election.cc:288] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:54.141202 13097 raft_consensus.cc:2781] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:54.141494 13097 raft_consensus.cc:482] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:54.141826 13097 raft_consensus.cc:3037] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.147526 13097 raft_consensus.cc:504] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.149169 13097 leader_election.cc:288] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 election: Requested vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:54.149825 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b31d21185422792582600382ca871" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:54.150120 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b31d21185422792582600382ca871" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:54.150288 12920 raft_consensus.cc:3037] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.150638 13028 raft_consensus.cc:3037] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.154587 12920 raft_consensus.cc:2454] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 1.
I20260807 07:59:54.155031 13028 raft_consensus.cc:2454] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 1.
I20260807 07:59:54.155488 12752 leader_election.cc:302] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:54.156121 13097 raft_consensus.cc:2781] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:54.156584 13097 raft_consensus.cc:686] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 1 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 07:59:54.157232 13097 consensus_queue.cc:237] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.160259 13139 raft_consensus.cc:482] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:54.160697 13139 raft_consensus.cc:504] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.162209 13139 leader_election.cc:288] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:54.163221 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:54.163297 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:54.163785 12920 raft_consensus.cc:2454] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 0.
I20260807 07:59:54.163890 12804 raft_consensus.cc:2454] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 0.
I20260807 07:59:54.164943 12977 leader_election.cc:302] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:54.165767 13139 raft_consensus.cc:2781] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:54.166170 13139 raft_consensus.cc:482] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:54.166533 13139 raft_consensus.cc:3037] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.166301 12656 catalog_manager.cc:5495] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 1 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:54.172588 13139 raft_consensus.cc:504] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.174365 13139 leader_election.cc:288] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:54.175122 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:54.175387 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:54.175719 12804 raft_consensus.cc:3037] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.175951 12920 raft_consensus.cc:3037] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.181962 12920 raft_consensus.cc:2454] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 1.
I20260807 07:59:54.181978 12804 raft_consensus.cc:2454] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 1.
I20260807 07:59:54.183073 12973 leader_election.cc:302] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:54.183703 13139 raft_consensus.cc:2781] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:54.184084 13139 raft_consensus.cc:686] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Becoming Leader. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 07:59:54.184847 13139 consensus_queue.cc:237] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.191304 12656 catalog_manager.cc:5495] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf602cc636547d59d67849689e3c1b7 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "aaf602cc636547d59d67849689e3c1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: HEALTHY } } }
W20260807 07:59:54.306910 13071 tablet.cc:2369] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 07:59:54.321197 13139 raft_consensus.cc:482] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:54.321745 13139 raft_consensus.cc:504] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.323467 13139 leader_election.cc:288] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:54.324424 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:54.324453 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:54.325207 12804 raft_consensus.cc:2379] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aaf602cc636547d59d67849689e3c1b7 in current term 1: Already voted for candidate 166256f885304c169d14729ac6311f16 in this term.
I20260807 07:59:54.326010 12977 leader_election.cc:302] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aaf602cc636547d59d67849689e3c1b7; no voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16
I20260807 07:59:54.326607 13139 raft_consensus.cc:3037] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.330900 13139 raft_consensus.cc:2726] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:54.348315 13097 consensus_queue.cc:1035] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 07:59:54.373258 13139 raft_consensus.cc:482] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:54.374007 13139 raft_consensus.cc:504] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.376255 13139 leader_election.cc:288] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:54.377869 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a532ddc55e4e421a9ab42b09bb326b41" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:54.378013 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a532ddc55e4e421a9ab42b09bb326b41" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:54.377928 13122 consensus_queue.cc:1035] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 07:59:54.378773 12804 raft_consensus.cc:2379] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aaf602cc636547d59d67849689e3c1b7 in current term 1: Already voted for candidate 166256f885304c169d14729ac6311f16 in this term.
I20260807 07:59:54.379844 12977 leader_election.cc:302] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aaf602cc636547d59d67849689e3c1b7; no voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16
I20260807 07:59:54.380506 13139 raft_consensus.cc:3037] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.387714 13139 raft_consensus.cc:2726] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 07:59:54.400918 13139 raft_consensus.cc:482] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:54.401327 13139 raft_consensus.cc:504] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.403687 13139 leader_election.cc:288] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:54.404639 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933df2ca62e24d5da058bb135d66eac4" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:54.404790 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933df2ca62e24d5da058bb135d66eac4" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:54.405416 12804 raft_consensus.cc:2454] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 0.
I20260807 07:59:54.405457 12920 raft_consensus.cc:2454] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 0.
I20260807 07:59:54.406904 12973 leader_election.cc:302] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:54.407812 13140 raft_consensus.cc:2781] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 07:59:54.408208 13140 raft_consensus.cc:482] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:54.408545 13140 raft_consensus.cc:3037] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.409459 13139 consensus_queue.cc:1035] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.415453 13140 raft_consensus.cc:504] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.418553 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933df2ca62e24d5da058bb135d66eac4" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:54.419149 12920 raft_consensus.cc:3037] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.418156 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933df2ca62e24d5da058bb135d66eac4" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:54.420655 12804 raft_consensus.cc:3037] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.426642 12920 raft_consensus.cc:2454] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 1.
I20260807 07:59:54.427692 12973 leader_election.cc:302] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:54.428664 13188 raft_consensus.cc:2781] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 07:59:54.428469 13187 consensus_queue.cc:1035] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.429024 12804 raft_consensus.cc:2454] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 1.
I20260807 07:59:54.429463 13188 raft_consensus.cc:686] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Becoming Leader. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 07:59:54.430330 13188 consensus_queue.cc:237] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:54.439800 13140 leader_election.cc:288] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 1 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:54.446462 13096 consensus_queue.cc:1035] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.446135 12656 catalog_manager.cc:5495] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf602cc636547d59d67849689e3c1b7 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "aaf602cc636547d59d67849689e3c1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:54.456441 13096 consensus_queue.cc:1035] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 07:59:54.478521 12804 raft_consensus.cc:3037] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 07:59:54.484337 13096 consensus_queue.cc:1035] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.495215 13096 consensus_queue.cc:1035] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.567366 13188 consensus_queue.cc:1035] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.577299 13187 consensus_queue.cc:1035] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.597237 13096 consensus_queue.cc:1035] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.607352 13142 consensus_queue.cc:1035] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 07:59:54.672942 12723 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 07:59:54.702198 13142 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.712325 13142 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.762481 13122 consensus_queue.cc:1035] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.772421 13097 consensus_queue.cc:1035] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.912079 13187 consensus_queue.cc:1035] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:54.923712 13140 consensus_queue.cc:1035] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:55.558687 13139 raft_consensus.cc:482] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 07:59:55.559114 13139 raft_consensus.cc:504] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:55.560526 13139 leader_election.cc:288] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:55.561420 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7" is_pre_election: true
I20260807 07:59:55.561610 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16" is_pre_election: true
I20260807 07:59:55.561992 12804 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 1.
I20260807 07:59:55.562176 12920 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 1.
I20260807 07:59:55.563060 12977 leader_election.cc:302] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:55.563853 13139 raft_consensus.cc:2781] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 07:59:55.564167 13139 raft_consensus.cc:482] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 07:59:55.564385 13139 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:55.569100 13139 raft_consensus.cc:504] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:55.570772 13139 leader_election.cc:288] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 2 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:55.571589 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:55.571949 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:55.572185 12804 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:55.572558 12920 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:55.577682 12804 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 2.
I20260807 07:59:55.578030 12920 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 2.
I20260807 07:59:55.578640 12977 leader_election.cc:302] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:55.579517 13139 raft_consensus.cc:2781] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:55.579988 13139 raft_consensus.cc:686] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Becoming Leader. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 07:59:55.580592 13139 consensus_queue.cc:237] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:55.587621 12660 catalog_manager.cc:5495] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 reported cstate change: term changed from 0 to 2, leader changed from <none> to aaf602cc636547d59d67849689e3c1b7 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "aaf602cc636547d59d67849689e3c1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:56.030236 13028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1"
dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
mode: GRACEFUL
new_leader_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38150
I20260807 07:59:56.030814 13028 raft_consensus.cc:595] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Received request to transfer leadership to TS 1100218421414a9fa7abd553db3cfca7
I20260807 07:59:56.043542 12920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da"
dest_uuid: "166256f885304c169d14729ac6311f16"
mode: GRACEFUL
new_leader_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:43228
I20260807 07:59:56.044032 12920 raft_consensus.cc:595] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Received request to transfer leadership to TS 1100218421414a9fa7abd553db3cfca7
I20260807 07:59:56.051451 13230 raft_consensus.cc:979] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16: : Instructing follower 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 07:59:56.051908 13230 raft_consensus.cc:1067] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Signalling peer 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 07:59:56.053340 12920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a068e349885444c89055c339659e95a5"
dest_uuid: "166256f885304c169d14729ac6311f16"
mode: GRACEFUL
new_leader_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:43228
I20260807 07:59:56.053726 12803 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38488
I20260807 07:59:56.053929 12920 raft_consensus.cc:595] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Received request to transfer leadership to TS 1100218421414a9fa7abd553db3cfca7
I20260807 07:59:56.054277 12803 raft_consensus.cc:482] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:56.054649 12803 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.057576 12804 raft_consensus.cc:1261] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 07:59:56.058988 13139 consensus_queue.cc:1035] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 07:59:56.060562 12803 raft_consensus.cc:504] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.062707 13028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55"
dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
mode: GRACEFUL
new_leader_uuid: "166256f885304c169d14729ac6311f16"
from {username='slave'} at 127.0.0.1:38150
I20260807 07:59:56.063338 13028 raft_consensus.cc:595] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Received request to transfer leadership to TS 166256f885304c169d14729ac6311f16
I20260807 07:59:56.063262 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:56.063835 12803 leader_election.cc:288] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 2 election: Requested vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:56.063889 12920 raft_consensus.cc:3032] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:56.064366 12920 raft_consensus.cc:729] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 07:59:56.065128 12920 consensus_queue.cc:260] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.066514 12920 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.072046 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:56.072955 13028 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.075367 12920 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 2.
I20260807 07:59:56.078413 13028 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 2.
I20260807 07:59:56.079800 12756 leader_election.cc:302] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:56.080135 12920 raft_consensus.cc:1261] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260807 07:59:56.080706 13211 raft_consensus.cc:2781] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:56.081972 13211 raft_consensus.cc:686] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 07:59:56.082707 13211 consensus_queue.cc:237] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.084757 13234 consensus_queue.cc:1035] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 07:59:56.090411 13028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "06b99890fa394a81853cfc4805b31172"
dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
mode: GRACEFUL
new_leader_uuid: "166256f885304c169d14729ac6311f16"
from {username='slave'} at 127.0.0.1:38150
I20260807 07:59:56.091231 13028 raft_consensus.cc:595] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Received request to transfer leadership to TS 166256f885304c169d14729ac6311f16
I20260807 07:59:56.103958 13234 raft_consensus.cc:979] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7: : Instructing follower 166256f885304c169d14729ac6311f16 to start an election
I20260807 07:59:56.104365 13140 raft_consensus.cc:1067] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Signalling peer 166256f885304c169d14729ac6311f16 to start an election
I20260807 07:59:56.106520 12920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55"
dest_uuid: "166256f885304c169d14729ac6311f16"
from {username='slave'} at 127.0.0.1:43204
I20260807 07:59:56.107076 12920 raft_consensus.cc:482] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:56.107369 12920 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:56.106670 12660 catalog_manager.cc:5495] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 1 to 2, leader changed from 166256f885304c169d14729ac6311f16 (127.11.140.66) to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 2 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:56.111644 12920 raft_consensus.cc:504] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.113467 12920 leader_election.cc:288] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 3 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:56.115020 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:56.115597 13028 raft_consensus.cc:3032] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 07:59:56.115918 13028 raft_consensus.cc:729] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 07:59:56.115813 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:56.116463 12804 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:56.116747 13028 consensus_queue.cc:260] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.118014 13028 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:56.121505 12804 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 3.
I20260807 07:59:56.123987 13028 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 3.
I20260807 07:59:56.124648 12864 leader_election.cc:302] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:56.125375 13230 raft_consensus.cc:2781] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:59:56.125851 13230 raft_consensus.cc:686] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 3 LEADER]: Becoming Leader. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 07:59:56.126542 13230 consensus_queue.cc:237] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.132313 12656 catalog_manager.cc:5495] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 reported cstate change: term changed from 2 to 3, leader changed from aaf602cc636547d59d67849689e3c1b7 (127.11.140.67) to 166256f885304c169d14729ac6311f16 (127.11.140.66). New cstate: current_term: 3 leader_uuid: "166256f885304c169d14729ac6311f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:56.205008 13234 raft_consensus.cc:979] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7: : Instructing follower 166256f885304c169d14729ac6311f16 to start an election
I20260807 07:59:56.205408 13140 raft_consensus.cc:1067] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Signalling peer 166256f885304c169d14729ac6311f16 to start an election
I20260807 07:59:56.206662 12920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06b99890fa394a81853cfc4805b31172"
dest_uuid: "166256f885304c169d14729ac6311f16"
from {username='slave'} at 127.0.0.1:43204
I20260807 07:59:56.207149 12920 raft_consensus.cc:482] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:56.207453 12920 raft_consensus.cc:3037] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.211786 12920 raft_consensus.cc:504] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.213183 12920 leader_election.cc:288] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 2 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:56.214063 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:56.214282 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:56.214664 12804 raft_consensus.cc:3037] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.214787 13028 raft_consensus.cc:3032] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:56.215121 13028 raft_consensus.cc:729] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 07:59:56.215865 13028 consensus_queue.cc:260] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.217038 13028 raft_consensus.cc:3037] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.220849 12804 raft_consensus.cc:2454] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 2.
I20260807 07:59:56.221868 12864 leader_election.cc:302] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:56.222541 13230 raft_consensus.cc:2781] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:56.223033 13230 raft_consensus.cc:686] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 2 LEADER]: Becoming Leader. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 07:59:56.223033 13028 raft_consensus.cc:2454] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 2.
I20260807 07:59:56.224018 13230 consensus_queue.cc:237] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.230374 12656 catalog_manager.cc:5495] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 reported cstate change: term changed from 1 to 2, leader changed from aaf602cc636547d59d67849689e3c1b7 (127.11.140.67) to 166256f885304c169d14729ac6311f16 (127.11.140.66). New cstate: current_term: 2 leader_uuid: "166256f885304c169d14729ac6311f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:56.293833 13167 raft_consensus.cc:979] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16: : Instructing follower 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 07:59:56.294291 13230 raft_consensus.cc:1067] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Signalling peer 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 07:59:56.295693 12804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a068e349885444c89055c339659e95a5"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38488
I20260807 07:59:56.296172 12804 raft_consensus.cc:482] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:56.296476 12804 raft_consensus.cc:3037] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.300760 12804 raft_consensus.cc:504] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.302408 12804 leader_election.cc:288] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 2 election: Requested vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 07:59:56.303176 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a068e349885444c89055c339659e95a5" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:56.303459 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a068e349885444c89055c339659e95a5" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:56.303685 12920 raft_consensus.cc:3032] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:56.304008 13028 raft_consensus.cc:3037] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.304016 12920 raft_consensus.cc:729] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 07:59:56.304847 12920 consensus_queue.cc:260] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.305792 12920 raft_consensus.cc:3037] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.308497 13028 raft_consensus.cc:2454] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 2.
I20260807 07:59:56.309381 12756 leader_election.cc:302] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:56.310102 13211 raft_consensus.cc:2781] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:56.310596 13211 raft_consensus.cc:686] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 07:59:56.310596 12920 raft_consensus.cc:2454] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 2.
I20260807 07:59:56.311535 13211 consensus_queue.cc:237] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:56.318385 12656 catalog_manager.cc:5495] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 1 to 2, leader changed from 166256f885304c169d14729ac6311f16 (127.11.140.66) to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 2 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:56.453756 13234 raft_consensus.cc:979] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7: : Instructing follower 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 07:59:56.454170 13140 raft_consensus.cc:1067] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Signalling peer 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 07:59:56.455355 12804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38478
I20260807 07:59:56.455842 12804 raft_consensus.cc:482] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:56.456136 12804 raft_consensus.cc:3037] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.460934 12804 raft_consensus.cc:504] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:56.462510 12804 leader_election.cc:288] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 2 election: Requested vote from peers aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:56.463212 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 07:59:56.463543 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:56.463811 13028 raft_consensus.cc:3032] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:56.464139 12920 raft_consensus.cc:3037] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.464193 13028 raft_consensus.cc:729] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 07:59:56.464937 13028 consensus_queue.cc:260] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:56.465925 13028 raft_consensus.cc:3037] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:56.468873 12920 raft_consensus.cc:2454] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 2.
I20260807 07:59:56.469807 12752 leader_election.cc:302] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 07:59:56.470470 13211 raft_consensus.cc:2781] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:56.470515 13028 raft_consensus.cc:2454] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 2.
I20260807 07:59:56.471066 13211 raft_consensus.cc:686] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 07:59:56.471891 13211 consensus_queue.cc:237] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 07:59:56.478304 12656 catalog_manager.cc:5495] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 1 to 2, leader changed from aaf602cc636547d59d67849689e3c1b7 (127.11.140.67) to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 2 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } }
I20260807 07:59:56.495514 13028 raft_consensus.cc:1261] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:56.496649 13211 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:56.505378 12920 raft_consensus.cc:1261] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:56.507097 13211 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:56.530422 13028 raft_consensus.cc:1261] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Refusing update from remote peer 166256f885304c169d14729ac6311f16: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 07:59:56.531561 13230 consensus_queue.cc:1035] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:56.538636 12804 raft_consensus.cc:1261] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Refusing update from remote peer 166256f885304c169d14729ac6311f16: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 07:59:56.540509 13256 consensus_queue.cc:1035] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:56.673754 12804 raft_consensus.cc:1261] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Refusing update from remote peer 166256f885304c169d14729ac6311f16: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:56.674822 13256 consensus_queue.cc:1035] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:56.684652 13028 raft_consensus.cc:1261] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Refusing update from remote peer 166256f885304c169d14729ac6311f16: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:56.685998 13230 consensus_queue.cc:1035] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:56.857079 12920 raft_consensus.cc:1261] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:56.858368 13211 consensus_queue.cc:1035] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:56.866379 13029 raft_consensus.cc:1261] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:56.867825 13253 consensus_queue.cc:1035] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:56.905747 12920 raft_consensus.cc:1261] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:56.906924 13253 consensus_queue.cc:1035] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:56.917325 13029 raft_consensus.cc:1261] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:56.918668 13253 consensus_queue.cc:1035] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:59.127416 11825 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260807 07:59:59.128078 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 07:59:59.148144 12804 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73cf365291be4223bf9744b896d0743d"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
mode: GRACEFUL
new_leader_uuid: "aaf602cc636547d59d67849689e3c1b7"
from {username='slave'} at 127.0.0.1:38490
I20260807 07:59:59.148623 12804 raft_consensus.cc:595] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 1 LEADER]: Received request to transfer leadership to TS aaf602cc636547d59d67849689e3c1b7
I20260807 07:59:59.150959 12804 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
mode: GRACEFUL
new_leader_uuid: "aaf602cc636547d59d67849689e3c1b7"
from {username='slave'} at 127.0.0.1:38490
I20260807 07:59:59.151517 12804 raft_consensus.cc:595] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Received request to transfer leadership to TS aaf602cc636547d59d67849689e3c1b7
I20260807 07:59:59.152472 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260807 07:59:59.153054 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 07:59:59.336948 13236 raft_consensus.cc:979] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7: : Instructing follower aaf602cc636547d59d67849689e3c1b7 to start an election
I20260807 07:59:59.337389 13271 raft_consensus.cc:1067] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Signalling peer aaf602cc636547d59d67849689e3c1b7 to start an election
I20260807 07:59:59.338790 13028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da"
dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
from {username='slave'} at 127.0.0.1:38134
I20260807 07:59:59.339233 13028 raft_consensus.cc:482] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:59.339490 13028 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:59.343559 13028 raft_consensus.cc:504] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:59.344792 13028 leader_election.cc:288] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 3 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:59.345654 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:59.345821 12922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:59.346150 12804 raft_consensus.cc:3032] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 07:59:59.346342 12922 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:59.346417 12804 raft_consensus.cc:729] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 07:59:59.347129 12804 consensus_queue.cc:260] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:59.347930 12804 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 07:59:59.350893 12922 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 3.
I20260807 07:59:59.351802 12804 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 3.
I20260807 07:59:59.351869 12973 leader_election.cc:302] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:59.352617 13140 raft_consensus.cc:2781] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 07:59:59.353654 13140 raft_consensus.cc:686] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 3 LEADER]: Becoming Leader. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 07:59:59.354296 13140 consensus_queue.cc:237] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:59.360821 12656 catalog_manager.cc:5495] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 reported cstate change: term changed from 2 to 3, leader changed from 1100218421414a9fa7abd553db3cfca7 (127.11.140.65) to aaf602cc636547d59d67849689e3c1b7 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "aaf602cc636547d59d67849689e3c1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:59.492102 13236 raft_consensus.cc:979] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7: : Instructing follower aaf602cc636547d59d67849689e3c1b7 to start an election
I20260807 07:59:59.492532 13271 raft_consensus.cc:1067] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 1 LEADER]: Signalling peer aaf602cc636547d59d67849689e3c1b7 to start an election
I20260807 07:59:59.493844 13028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73cf365291be4223bf9744b896d0743d"
dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
from {username='slave'} at 127.0.0.1:38134
I20260807 07:59:59.494223 13028 raft_consensus.cc:482] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 07:59:59.494428 13028 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:59.498482 13028 raft_consensus.cc:504] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:59.499735 13028 leader_election.cc:288] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 2 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 07:59:59.500743 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 07:59:59.500743 12922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 07:59:59.501495 12804 raft_consensus.cc:3032] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 07:59:59.501565 12922 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:59.501991 12804 raft_consensus.cc:729] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 07:59:59.502722 12804 consensus_queue.cc:260] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:59.503813 12804 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 07:59:59.507987 12922 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 2.
I20260807 07:59:59.508967 12973 leader_election.cc:302] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 07:59:59.509639 13140 raft_consensus.cc:2781] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 07:59:59.509599 12804 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 2.
I20260807 07:59:59.510232 13140 raft_consensus.cc:686] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Becoming Leader. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 07:59:59.511021 13140 consensus_queue.cc:237] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 07:59:59.518282 12656 catalog_manager.cc:5495] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 reported cstate change: term changed from 1 to 2, leader changed from 1100218421414a9fa7abd553db3cfca7 (127.11.140.65) to aaf602cc636547d59d67849689e3c1b7 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "aaf602cc636547d59d67849689e3c1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: HEALTHY } } }
I20260807 07:59:59.832000 12804 raft_consensus.cc:1261] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:59.833081 13140 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:59.843066 12922 raft_consensus.cc:1261] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 07:59:59.845664 13272 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 07:59:59.904192 12804 raft_consensus.cc:1261] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:59.905295 13272 consensus_queue.cc:1035] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 07:59:59.914106 12922 raft_consensus.cc:1261] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 07:59:59.915402 13140 consensus_queue.cc:1035] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:00.154000 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:00.159421 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:00.159761 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:01.204967 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:01.214910 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:01.215718 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:02.217341 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:02.223902 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:02.224274 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260807 08:00:02.240935 13064 debug-util.cc:398] Leaking SignalData structure 0x7b08000ee5e0 after lost signal to thread 11828
W20260807 08:00:02.242017 13064 debug-util.cc:398] Leaking SignalData structure 0x7b08000eecc0 after lost signal to thread 12705
W20260807 08:00:02.243461 13064 debug-util.cc:398] Leaking SignalData structure 0x7b080014a7a0 after lost signal to thread 12840
W20260807 08:00:02.244791 13064 debug-util.cc:398] Leaking SignalData structure 0x7b0800084de0 after lost signal to thread 12957
W20260807 08:00:02.245975 13064 debug-util.cc:398] Leaking SignalData structure 0x7b08001fe380 after lost signal to thread 13068
I20260807 08:00:03.225116 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:03.230594 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:03.231073 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:04.231961 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:04.237926 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:04.238354 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:05.239310 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:05.245617 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:05.245975 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:06.246735 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:06.252099 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:06.252524 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:07.253347 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:07.258989 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:07.259411 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:08.260190 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:08.266539 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:08.267027 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:09.296265 11825 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260807 08:00:09.302743 13029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da"
dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
mode: GRACEFUL
new_leader_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38150
I20260807 08:00:09.303465 13029 raft_consensus.cc:595] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 3 LEADER]: Received request to transfer leadership to TS 1100218421414a9fa7abd553db3cfca7
I20260807 08:00:09.310281 13028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73cf365291be4223bf9744b896d0743d"
dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
mode: GRACEFUL
new_leader_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38150
I20260807 08:00:09.310752 13028 raft_consensus.cc:595] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Received request to transfer leadership to TS 1100218421414a9fa7abd553db3cfca7
I20260807 08:00:09.317368 12920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7"
dest_uuid: "166256f885304c169d14729ac6311f16"
mode: GRACEFUL
new_leader_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:43228
I20260807 08:00:09.317888 12920 raft_consensus.cc:595] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Received request to transfer leadership to TS 1100218421414a9fa7abd553db3cfca7
I20260807 08:00:09.322098 12920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55"
dest_uuid: "166256f885304c169d14729ac6311f16"
mode: GRACEFUL
new_leader_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:43228
I20260807 08:00:09.322589 12920 raft_consensus.cc:595] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 3 LEADER]: Received request to transfer leadership to TS 1100218421414a9fa7abd553db3cfca7
I20260807 08:00:09.326560 12920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "06b99890fa394a81853cfc4805b31172"
dest_uuid: "166256f885304c169d14729ac6311f16"
mode: GRACEFUL
new_leader_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:43228
I20260807 08:00:09.326993 12920 raft_consensus.cc:595] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 2 LEADER]: Received request to transfer leadership to TS 1100218421414a9fa7abd553db3cfca7
I20260807 08:00:09.364398 13300 raft_consensus.cc:979] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16: : Instructing follower 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.365011 13305 raft_consensus.cc:1067] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 3 LEADER]: Signalling peer 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.366488 12804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38488
I20260807 08:00:09.366909 12804 raft_consensus.cc:482] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:09.367164 12804 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:09.373232 12804 raft_consensus.cc:504] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.375536 12804 leader_election.cc:288] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 4 election: Requested vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 08:00:09.376377 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 08:00:09.376660 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7392a5e0939b4d7d92f3ead5b9f05a55" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 08:00:09.377152 12920 raft_consensus.cc:3032] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:00:09.377244 13028 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:09.377496 12920 raft_consensus.cc:729] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 08:00:09.378464 12920 consensus_queue.cc:260] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.379884 12920 raft_consensus.cc:3037] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:09.382529 13028 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 4.
I20260807 08:00:09.383611 12756 leader_election.cc:302] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 08:00:09.384380 13304 raft_consensus.cc:2781] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:00:09.384871 13304 raft_consensus.cc:686] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 4 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 08:00:09.384856 12920 raft_consensus.cc:2454] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 4.
I20260807 08:00:09.385957 13304 consensus_queue.cc:237] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.398052 12660 catalog_manager.cc:5495] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 3 to 4, leader changed from 166256f885304c169d14729ac6311f16 (127.11.140.66) to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 4 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:09.512697 13300 raft_consensus.cc:979] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16: : Instructing follower 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.513221 13305 raft_consensus.cc:1067] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Signalling peer 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.514593 12804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38488
I20260807 08:00:09.515161 12804 raft_consensus.cc:482] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:09.515439 12804 raft_consensus.cc:3037] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:09.519757 12804 raft_consensus.cc:504] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.521162 12804 leader_election.cc:288] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 2 election: Requested vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 08:00:09.522099 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 08:00:09.522150 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca6eb08998348c88b1e71c8d0dc71e7" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 08:00:09.522624 13028 raft_consensus.cc:3037] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:09.522692 12920 raft_consensus.cc:3032] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:09.523052 12920 raft_consensus.cc:729] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 08:00:09.523630 12920 consensus_queue.cc:260] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.524474 12920 raft_consensus.cc:3037] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:09.526727 13028 raft_consensus.cc:2454] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 2.
I20260807 08:00:09.527487 12756 leader_election.cc:302] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 08:00:09.528087 13304 raft_consensus.cc:2781] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:09.528530 13304 raft_consensus.cc:686] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 08:00:09.529173 12920 raft_consensus.cc:2454] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 2.
I20260807 08:00:09.529414 13304 consensus_queue.cc:237] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.534941 13140 raft_consensus.cc:979] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7: : Instructing follower 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.535434 13313 raft_consensus.cc:1067] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Signalling peer 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.536742 12804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73cf365291be4223bf9744b896d0743d"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38478
I20260807 08:00:09.536573 12660 catalog_manager.cc:5495] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 1 to 2, leader changed from 166256f885304c169d14729ac6311f16 (127.11.140.66) to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 2 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:09.537155 12804 raft_consensus.cc:482] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:09.537460 12804 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:09.541399 12804 raft_consensus.cc:504] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.543188 12804 leader_election.cc:288] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 3 election: Requested vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 08:00:09.543550 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 08:00:09.544070 12920 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:09.544384 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 08:00:09.544759 13028 raft_consensus.cc:3032] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:00:09.544924 13028 raft_consensus.cc:729] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 08:00:09.545614 13028 consensus_queue.cc:260] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.546450 13028 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:09.547953 12920 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 3.
I20260807 08:00:09.548810 12752 leader_election.cc:302] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 08:00:09.549458 13304 raft_consensus.cc:2781] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:00:09.549907 13304 raft_consensus.cc:686] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 3 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 08:00:09.550967 13028 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 3.
I20260807 08:00:09.550727 13304 consensus_queue.cc:237] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.552934 13300 raft_consensus.cc:979] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16: : Instructing follower 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.553377 13305 raft_consensus.cc:1067] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 2 LEADER]: Signalling peer 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.555924 12804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06b99890fa394a81853cfc4805b31172"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38488
I20260807 08:00:09.556339 12804 raft_consensus.cc:482] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:09.556634 12804 raft_consensus.cc:3037] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:09.558285 12660 catalog_manager.cc:5495] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 2 to 3, leader changed from aaf602cc636547d59d67849689e3c1b7 (127.11.140.67) to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 3 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:09.561995 12804 raft_consensus.cc:504] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.563688 12804 leader_election.cc:288] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 3 election: Requested vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 08:00:09.564113 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 08:00:09.564236 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b99890fa394a81853cfc4805b31172" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 08:00:09.564706 12920 raft_consensus.cc:3032] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:00:09.564762 13028 raft_consensus.cc:3037] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:09.565057 12920 raft_consensus.cc:729] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 08:00:09.565951 12920 consensus_queue.cc:260] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.567530 12920 raft_consensus.cc:3037] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:09.569099 13028 raft_consensus.cc:2454] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 3.
I20260807 08:00:09.569905 12756 leader_election.cc:302] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 08:00:09.570525 13304 raft_consensus.cc:2781] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:00:09.570966 13304 raft_consensus.cc:686] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 3 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 08:00:09.572139 12920 raft_consensus.cc:2454] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 3.
I20260807 08:00:09.571749 13304 consensus_queue.cc:237] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.578152 12660 catalog_manager.cc:5495] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 2 to 3, leader changed from 166256f885304c169d14729ac6311f16 (127.11.140.66) to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 3 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:09.723580 13140 raft_consensus.cc:979] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7: : Instructing follower 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.724005 13313 raft_consensus.cc:1067] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 3 LEADER]: Signalling peer 1100218421414a9fa7abd553db3cfca7 to start an election
I20260807 08:00:09.725404 12804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
from {username='slave'} at 127.0.0.1:38478
I20260807 08:00:09.725859 12804 raft_consensus.cc:482] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:09.726161 12804 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:09.730340 12804 raft_consensus.cc:504] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.731657 12804 leader_election.cc:288] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 4 election: Requested vote from peers 166256f885304c169d14729ac6311f16 (127.11.140.66:45069), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 08:00:09.732488 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 08:00:09.732617 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "1100218421414a9fa7abd553db3cfca7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 08:00:09.733057 12920 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:09.733125 13028 raft_consensus.cc:3032] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:00:09.733420 13028 raft_consensus.cc:729] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 08:00:09.734185 13028 consensus_queue.cc:260] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.735277 13028 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:09.739114 12920 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 4.
I20260807 08:00:09.740074 12752 leader_election.cc:302] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100218421414a9fa7abd553db3cfca7, 166256f885304c169d14729ac6311f16; no voters:
I20260807 08:00:09.740787 13304 raft_consensus.cc:2781] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:00:09.740797 13028 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100218421414a9fa7abd553db3cfca7 in term 4.
I20260807 08:00:09.741272 13304 raft_consensus.cc:686] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 4 LEADER]: Becoming Leader. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 08:00:09.742110 13304 consensus_queue.cc:237] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:09.748783 12660 catalog_manager.cc:5495] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 reported cstate change: term changed from 3 to 4, leader changed from aaf602cc636547d59d67849689e3c1b7 (127.11.140.67) to 1100218421414a9fa7abd553db3cfca7 (127.11.140.65). New cstate: current_term: 4 leader_uuid: "1100218421414a9fa7abd553db3cfca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:09.880398 13028 raft_consensus.cc:1261] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:00:09.881698 13304 consensus_queue.cc:1035] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:09.891913 12920 raft_consensus.cc:1261] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:00:09.893430 13304 consensus_queue.cc:1035] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:09.931855 13028 raft_consensus.cc:1261] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:09.933085 13319 consensus_queue.cc:1035] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:09.941285 12920 raft_consensus.cc:1261] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:09.942852 13304 consensus_queue.cc:1035] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:09.950103 12920 raft_consensus.cc:1261] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:09.951332 13304 consensus_queue.cc:1035] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:09.959764 13028 raft_consensus.cc:1261] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:09.961071 13303 consensus_queue.cc:1035] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:09.972798 13029 raft_consensus.cc:1261] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:09.974368 13303 consensus_queue.cc:1035] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:09.985327 12920 raft_consensus.cc:1261] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:09.986708 13303 consensus_queue.cc:1035] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:10.156358 12920 raft_consensus.cc:1261] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:00:10.157678 13303 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:10.166038 13028 raft_consensus.cc:1261] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Refusing update from remote peer 1100218421414a9fa7abd553db3cfca7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:00:10.167701 13304 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:12.360723 11825 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260807 08:00:12.361538 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:12.368608 12804 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
mode: GRACEFUL
new_leader_uuid: "aaf602cc636547d59d67849689e3c1b7"
from {username='slave'} at 127.0.0.1:38490
I20260807 08:00:12.369390 12804 raft_consensus.cc:595] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Received request to transfer leadership to TS aaf602cc636547d59d67849689e3c1b7
I20260807 08:00:12.371994 12804 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73cf365291be4223bf9744b896d0743d"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
mode: GRACEFUL
new_leader_uuid: "166256f885304c169d14729ac6311f16"
from {username='slave'} at 127.0.0.1:38490
I20260807 08:00:12.372478 12804 raft_consensus.cc:595] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 3 LEADER]: Received request to transfer leadership to TS 166256f885304c169d14729ac6311f16
I20260807 08:00:12.374579 12804 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a068e349885444c89055c339659e95a5"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
mode: GRACEFUL
new_leader_uuid: "aaf602cc636547d59d67849689e3c1b7"
from {username='slave'} at 127.0.0.1:38490
I20260807 08:00:12.375016 12804 raft_consensus.cc:595] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Received request to transfer leadership to TS aaf602cc636547d59d67849689e3c1b7
I20260807 08:00:12.376869 12804 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da"
dest_uuid: "1100218421414a9fa7abd553db3cfca7"
mode: GRACEFUL
new_leader_uuid: "166256f885304c169d14729ac6311f16"
from {username='slave'} at 127.0.0.1:38490
I20260807 08:00:12.377295 12804 raft_consensus.cc:595] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 4 LEADER]: Received request to transfer leadership to TS 166256f885304c169d14729ac6311f16
I20260807 08:00:12.378163 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260807 08:00:12.378610 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:12.531635 13341 raft_consensus.cc:979] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7: : Instructing follower 166256f885304c169d14729ac6311f16 to start an election
I20260807 08:00:12.532251 13344 raft_consensus.cc:1067] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 3 LEADER]: Signalling peer 166256f885304c169d14729ac6311f16 to start an election
I20260807 08:00:12.533946 12920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73cf365291be4223bf9744b896d0743d"
dest_uuid: "166256f885304c169d14729ac6311f16"
from {username='slave'} at 127.0.0.1:43216
I20260807 08:00:12.534516 12920 raft_consensus.cc:482] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:12.534996 12920 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:12.540573 12920 raft_consensus.cc:504] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:12.542649 12920 leader_election.cc:288] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 4 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 08:00:12.543476 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 08:00:12.543795 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf365291be4223bf9744b896d0743d" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 08:00:12.544317 12804 raft_consensus.cc:3032] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:00:12.544394 13028 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:12.544615 12804 raft_consensus.cc:729] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 08:00:12.545430 12804 consensus_queue.cc:260] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:12.546588 12804 raft_consensus.cc:3037] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:12.549322 13028 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 4.
I20260807 08:00:12.550477 12864 leader_election.cc:302] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 08:00:12.551261 13305 raft_consensus.cc:2781] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:00:12.551748 12804 raft_consensus.cc:2454] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 4.
I20260807 08:00:12.552325 13305 raft_consensus.cc:686] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 4 LEADER]: Becoming Leader. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 08:00:12.553192 13305 consensus_queue.cc:237] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:12.560724 12660 catalog_manager.cc:5495] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 reported cstate change: term changed from 3 to 4, leader changed from 1100218421414a9fa7abd553db3cfca7 (127.11.140.65) to 166256f885304c169d14729ac6311f16 (127.11.140.66). New cstate: current_term: 4 leader_uuid: "166256f885304c169d14729ac6311f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:12.629284 13344 raft_consensus.cc:979] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7: : Instructing follower 166256f885304c169d14729ac6311f16 to start an election
I20260807 08:00:12.629837 13341 raft_consensus.cc:1067] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 4 LEADER]: Signalling peer 166256f885304c169d14729ac6311f16 to start an election
I20260807 08:00:12.631135 12920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da"
dest_uuid: "166256f885304c169d14729ac6311f16"
from {username='slave'} at 127.0.0.1:43216
I20260807 08:00:12.631536 12920 raft_consensus.cc:482] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:12.631748 12920 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:00:12.636194 12920 raft_consensus.cc:504] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:12.637771 12920 leader_election.cc:288] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 5 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), aaf602cc636547d59d67849689e3c1b7 (127.11.140.67:46211)
I20260807 08:00:12.638427 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 08:00:12.638710 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186ea2e51aa4dc6b851ce39ba4a37da" candidate_uuid: "166256f885304c169d14729ac6311f16" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
I20260807 08:00:12.638970 12804 raft_consensus.cc:3032] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:00:12.639292 12804 raft_consensus.cc:729] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 08:00:12.639261 13028 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:00:12.640122 12804 consensus_queue.cc:260] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:12.641088 12804 raft_consensus.cc:3037] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:00:12.643939 13028 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 5.
I20260807 08:00:12.644841 12864 leader_election.cc:302] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 08:00:12.645427 12804 raft_consensus.cc:2454] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166256f885304c169d14729ac6311f16 in term 5.
I20260807 08:00:12.645550 13305 raft_consensus.cc:2781] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:00:12.646080 13305 raft_consensus.cc:686] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 5 LEADER]: Becoming Leader. State: Replica: 166256f885304c169d14729ac6311f16, State: Running, Role: LEADER
I20260807 08:00:12.646950 13305 consensus_queue.cc:237] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:12.653700 12660 catalog_manager.cc:5495] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 reported cstate change: term changed from 4 to 5, leader changed from 1100218421414a9fa7abd553db3cfca7 (127.11.140.65) to 166256f885304c169d14729ac6311f16 (127.11.140.66). New cstate: current_term: 5 leader_uuid: "166256f885304c169d14729ac6311f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:12.822212 13344 raft_consensus.cc:979] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7: : Instructing follower aaf602cc636547d59d67849689e3c1b7 to start an election
I20260807 08:00:12.822710 13341 raft_consensus.cc:1067] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Signalling peer aaf602cc636547d59d67849689e3c1b7 to start an election
I20260807 08:00:12.824115 13028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1"
dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
from {username='slave'} at 127.0.0.1:38134
I20260807 08:00:12.824558 13028 raft_consensus.cc:482] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:12.824828 13028 raft_consensus.cc:3037] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:12.829135 13028 raft_consensus.cc:504] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 08:00:12.830863 13028 leader_election.cc:288] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 3 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 08:00:12.831732 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 08:00:12.832067 12922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526cbf5f066347a885b5b2ef31f681d1" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 08:00:12.832221 12804 raft_consensus.cc:3032] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:00:12.832589 12922 raft_consensus.cc:3037] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:12.832605 12804 raft_consensus.cc:729] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 08:00:12.833377 12804 consensus_queue.cc:260] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 08:00:12.834337 12804 raft_consensus.cc:3037] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:12.837720 12922 raft_consensus.cc:2454] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 3.
I20260807 08:00:12.838611 12804 raft_consensus.cc:2454] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 3.
I20260807 08:00:12.838793 12973 leader_election.cc:302] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 08:00:12.839686 13313 raft_consensus.cc:2781] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:00:12.840766 13313 raft_consensus.cc:686] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 3 LEADER]: Becoming Leader. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 08:00:12.841679 13313 consensus_queue.cc:237] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } }
I20260807 08:00:12.848594 12660 catalog_manager.cc:5495] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 reported cstate change: term changed from 2 to 3, leader changed from 1100218421414a9fa7abd553db3cfca7 (127.11.140.65) to aaf602cc636547d59d67849689e3c1b7 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "aaf602cc636547d59d67849689e3c1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:12.897980 13344 raft_consensus.cc:979] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7: : Instructing follower aaf602cc636547d59d67849689e3c1b7 to start an election
I20260807 08:00:12.898453 13341 raft_consensus.cc:1067] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Signalling peer aaf602cc636547d59d67849689e3c1b7 to start an election
I20260807 08:00:12.899893 13028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a068e349885444c89055c339659e95a5"
dest_uuid: "aaf602cc636547d59d67849689e3c1b7"
from {username='slave'} at 127.0.0.1:38134
I20260807 08:00:12.900347 13028 raft_consensus.cc:482] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:12.900633 13028 raft_consensus.cc:3037] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:12.905102 13028 raft_consensus.cc:504] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:12.906556 13028 leader_election.cc:288] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 3 election: Requested vote from peers 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141), 166256f885304c169d14729ac6311f16 (127.11.140.66:45069)
I20260807 08:00:12.907231 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a068e349885444c89055c339659e95a5" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1100218421414a9fa7abd553db3cfca7"
I20260807 08:00:12.907477 12922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a068e349885444c89055c339659e95a5" candidate_uuid: "aaf602cc636547d59d67849689e3c1b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "166256f885304c169d14729ac6311f16"
I20260807 08:00:12.907768 12804 raft_consensus.cc:3032] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:00:12.908073 12804 raft_consensus.cc:729] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1100218421414a9fa7abd553db3cfca7, State: Running, Role: LEADER
I20260807 08:00:12.908099 12922 raft_consensus.cc:3037] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:12.908939 12804 consensus_queue.cc:260] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:12.909901 12804 raft_consensus.cc:3037] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:12.913707 12922 raft_consensus.cc:2454] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 3.
I20260807 08:00:12.914868 12973 leader_election.cc:302] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166256f885304c169d14729ac6311f16, aaf602cc636547d59d67849689e3c1b7; no voters:
I20260807 08:00:12.915148 12804 raft_consensus.cc:2454] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf602cc636547d59d67849689e3c1b7 in term 3.
I20260807 08:00:12.915541 13313 raft_consensus.cc:2781] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:00:12.916033 13313 raft_consensus.cc:686] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 3 LEADER]: Becoming Leader. State: Replica: aaf602cc636547d59d67849689e3c1b7, State: Running, Role: LEADER
I20260807 08:00:12.916778 13313 consensus_queue.cc:237] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } }
I20260807 08:00:12.923800 12660 catalog_manager.cc:5495] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 reported cstate change: term changed from 2 to 3, leader changed from 1100218421414a9fa7abd553db3cfca7 (127.11.140.65) to aaf602cc636547d59d67849689e3c1b7 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "aaf602cc636547d59d67849689e3c1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:12.970999 12804 raft_consensus.cc:1261] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Refusing update from remote peer 166256f885304c169d14729ac6311f16: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:00:12.972172 13305 consensus_queue.cc:1035] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:12.981626 13028 raft_consensus.cc:1261] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Refusing update from remote peer 166256f885304c169d14729ac6311f16: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:00:12.983664 13305 consensus_queue.cc:1035] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:13.130308 12804 raft_consensus.cc:1261] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 5 FOLLOWER]: Refusing update from remote peer 166256f885304c169d14729ac6311f16: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 08:00:13.131525 13346 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:13.139662 13028 raft_consensus.cc:1261] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 5 FOLLOWER]: Refusing update from remote peer 166256f885304c169d14729ac6311f16: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260807 08:00:13.141429 13369 consensus_queue.cc:1035] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf602cc636547d59d67849689e3c1b7" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:00:13.259778 12922 raft_consensus.cc:1261] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:13.260885 13313 consensus_queue.cc:1035] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:13.269575 12804 raft_consensus.cc:1261] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:13.271026 13313 consensus_queue.cc:1035] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:13.379416 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:13.385232 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:13.385643 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:13.409154 12804 raft_consensus.cc:1261] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:13.410439 13355 consensus_queue.cc:1035] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100218421414a9fa7abd553db3cfca7" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:13.421152 12922 raft_consensus.cc:1261] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Refusing update from remote peer aaf602cc636547d59d67849689e3c1b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:13.422389 13355 consensus_queue.cc:1035] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166256f885304c169d14729ac6311f16" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:14.386395 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:14.392143 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:14.392493 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:15.393298 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:15.398826 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:15.399224 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:16.399988 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:16.405237 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:16.405669 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:17.406512 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:17.412389 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:17.412792 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:18.413661 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:18.418706 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:18.419157 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:19.419986 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:19.426193 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:19.426702 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:20.427582 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:20.432554 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:20.432945 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:21.433753 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:21.438798 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260807 08:00:21.439208 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:22.468516 11825 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260807 08:00:22.469161 11825 tablet_server.cc:178] TabletServer@127.11.140.65:0 shutting down...
I20260807 08:00:22.502602 11825 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 08:00:22.503276 11825 tablet_replica.cc:330] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.503954 11825 raft_consensus.cc:2229] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.504575 11825 raft_consensus.cc:2258] T a068e349885444c89055c339659e95a5 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.506722 11825 tablet_replica.cc:330] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.507270 11825 raft_consensus.cc:2229] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.508131 11825 raft_consensus.cc:2258] T fd1b31d21185422792582600382ca871 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.510004 11825 tablet_replica.cc:330] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.510504 11825 raft_consensus.cc:2229] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 3 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.511334 11825 raft_consensus.cc:2258] T 06b99890fa394a81853cfc4805b31172 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.513319 11825 tablet_replica.cc:330] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.513929 11825 raft_consensus.cc:2229] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 4 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.514732 11825 raft_consensus.cc:2258] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.516755 11825 tablet_replica.cc:330] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.517383 11825 raft_consensus.cc:2229] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.517992 11825 raft_consensus.cc:2258] T e186ea2e51aa4dc6b851ce39ba4a37da P 1100218421414a9fa7abd553db3cfca7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.520052 11825 tablet_replica.cc:330] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.520571 11825 raft_consensus.cc:2229] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 2 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.521389 11825 raft_consensus.cc:2258] T eca6eb08998348c88b1e71c8d0dc71e7 P 1100218421414a9fa7abd553db3cfca7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.523150 11825 tablet_replica.cc:330] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.523634 11825 raft_consensus.cc:2229] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.524039 11825 raft_consensus.cc:2258] T 933df2ca62e24d5da058bb135d66eac4 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.525735 11825 tablet_replica.cc:330] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.526186 11825 raft_consensus.cc:2229] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.526597 11825 raft_consensus.cc:2258] T a532ddc55e4e421a9ab42b09bb326b41 P 1100218421414a9fa7abd553db3cfca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.528240 11825 tablet_replica.cc:330] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.528688 11825 raft_consensus.cc:2229] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.529110 11825 raft_consensus.cc:2258] T 73cf365291be4223bf9744b896d0743d P 1100218421414a9fa7abd553db3cfca7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.530768 11825 tablet_replica.cc:330] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7: stopping tablet replica
I20260807 08:00:22.531193 11825 raft_consensus.cc:2229] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.531653 11825 raft_consensus.cc:2258] T 526cbf5f066347a885b5b2ef31f681d1 P 1100218421414a9fa7abd553db3cfca7 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260807 08:00:22.534809 12977 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20260807 08:00:22.538213 12977 consensus_peers.cc:487] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 -> Peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Couldn't send request to peer 1100218421414a9fa7abd553db3cfca7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:22.562476 11825 tablet_server.cc:194] TabletServer@127.11.140.65:0 shutdown complete.
W20260807 08:00:22.568523 12864 consensus_peers.cc:487] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 -> Peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Couldn't send request to peer 1100218421414a9fa7abd553db3cfca7. Status: Network error: Client connection negotiation failed: client connection to 127.11.140.65:33141: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:22.583588 11825 tablet_server.cc:178] TabletServer@127.11.140.66:0 shutting down...
I20260807 08:00:22.612568 11825 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 08:00:22.613384 11825 tablet_replica.cc:330] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16: stopping tablet replica
W20260807 08:00:22.613708 12864 consensus_peers.cc:487] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 -> Peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Couldn't send request to peer 1100218421414a9fa7abd553db3cfca7. Status: Network error: Client connection negotiation failed: client connection to 127.11.140.65:33141: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:22.614293 11825 raft_consensus.cc:2229] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.614809 11825 raft_consensus.cc:2258] T 526cbf5f066347a885b5b2ef31f681d1 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.616601 11825 tablet_replica.cc:330] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16: stopping tablet replica
I20260807 08:00:22.617101 11825 raft_consensus.cc:2229] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 5 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.617888 11825 raft_consensus.cc:2258] T e186ea2e51aa4dc6b851ce39ba4a37da P 166256f885304c169d14729ac6311f16 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.619529 11825 tablet_replica.cc:330] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16: stopping tablet replica
I20260807 08:00:22.619962 11825 raft_consensus.cc:2229] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.620301 11825 raft_consensus.cc:2258] T fd1b31d21185422792582600382ca871 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.621862 11825 tablet_replica.cc:330] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16: stopping tablet replica
I20260807 08:00:22.622391 11825 raft_consensus.cc:2229] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.622782 11825 raft_consensus.cc:2258] T a068e349885444c89055c339659e95a5 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.624210 11825 tablet_replica.cc:330] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16: stopping tablet replica
I20260807 08:00:22.624646 11825 raft_consensus.cc:2229] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.624989 11825 raft_consensus.cc:2258] T eca6eb08998348c88b1e71c8d0dc71e7 P 166256f885304c169d14729ac6311f16 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.626613 11825 tablet_replica.cc:330] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16: stopping tablet replica
I20260807 08:00:22.627030 11825 raft_consensus.cc:2229] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.627394 11825 raft_consensus.cc:2258] T 7392a5e0939b4d7d92f3ead5b9f05a55 P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.628947 11825 tablet_replica.cc:330] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16: stopping tablet replica
I20260807 08:00:22.629426 11825 raft_consensus.cc:2229] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.629943 11825 raft_consensus.cc:2258] T 933df2ca62e24d5da058bb135d66eac4 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.631783 11825 tablet_replica.cc:330] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16: stopping tablet replica
I20260807 08:00:22.632299 11825 raft_consensus.cc:2229] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.633061 11825 raft_consensus.cc:2258] T a532ddc55e4e421a9ab42b09bb326b41 P 166256f885304c169d14729ac6311f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.634652 11825 tablet_replica.cc:330] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16: stopping tablet replica
I20260807 08:00:22.635108 11825 raft_consensus.cc:2229] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.635528 11825 raft_consensus.cc:2258] T 06b99890fa394a81853cfc4805b31172 P 166256f885304c169d14729ac6311f16 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.636996 11825 tablet_replica.cc:330] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16: stopping tablet replica
I20260807 08:00:22.637428 11825 raft_consensus.cc:2229] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 4 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.638270 11825 raft_consensus.cc:2258] T 73cf365291be4223bf9744b896d0743d P 166256f885304c169d14729ac6311f16 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.666384 11825 tablet_server.cc:194] TabletServer@127.11.140.66:0 shutdown complete.
W20260807 08:00:22.676170 12973 consensus_peers.cc:487] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 -> Peer 166256f885304c169d14729ac6311f16 (127.11.140.66:45069): Couldn't send request to peer 166256f885304c169d14729ac6311f16. Status: Network error: Client connection negotiation failed: client connection to 127.11.140.66:45069: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:22.687143 11825 tablet_server.cc:178] TabletServer@127.11.140.67:0 shutting down...
I20260807 08:00:22.712831 11825 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260807 08:00:22.713508 11825 tablet_replica.cc:330] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
I20260807 08:00:22.714254 11825 raft_consensus.cc:2229] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.714776 11825 raft_consensus.cc:2258] T 06b99890fa394a81853cfc4805b31172 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.717245 11825 tablet_replica.cc:330] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
W20260807 08:00:22.718034 12977 consensus_peers.cc:487] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 -> Peer 1100218421414a9fa7abd553db3cfca7 (127.11.140.65:33141): Couldn't send request to peer 1100218421414a9fa7abd553db3cfca7. Status: Network error: Client connection negotiation failed: client connection to 127.11.140.65:33141: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:22.718122 11825 raft_consensus.cc:2229] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.718663 11825 raft_consensus.cc:2258] T eca6eb08998348c88b1e71c8d0dc71e7 P aaf602cc636547d59d67849689e3c1b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.720293 11825 tablet_replica.cc:330] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
I20260807 08:00:22.720784 11825 raft_consensus.cc:2229] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.721227 11825 raft_consensus.cc:2258] T e186ea2e51aa4dc6b851ce39ba4a37da P aaf602cc636547d59d67849689e3c1b7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.722841 11825 tablet_replica.cc:330] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
I20260807 08:00:22.723306 11825 raft_consensus.cc:2229] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.723706 11825 raft_consensus.cc:2258] T a532ddc55e4e421a9ab42b09bb326b41 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.725154 11825 tablet_replica.cc:330] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
I20260807 08:00:22.725679 11825 raft_consensus.cc:2229] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.726411 11825 raft_consensus.cc:2258] T 933df2ca62e24d5da058bb135d66eac4 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.727949 11825 tablet_replica.cc:330] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
I20260807 08:00:22.728425 11825 raft_consensus.cc:2229] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.728886 11825 raft_consensus.cc:2258] T 7392a5e0939b4d7d92f3ead5b9f05a55 P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.730504 11825 tablet_replica.cc:330] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
I20260807 08:00:22.730966 11825 raft_consensus.cc:2229] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 3 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.731719 11825 raft_consensus.cc:2258] T a068e349885444c89055c339659e95a5 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.733243 11825 tablet_replica.cc:330] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
I20260807 08:00:22.733767 11825 raft_consensus.cc:2229] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.734205 11825 raft_consensus.cc:2258] T 73cf365291be4223bf9744b896d0743d P aaf602cc636547d59d67849689e3c1b7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.735950 11825 tablet_replica.cc:330] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
I20260807 08:00:22.736407 11825 raft_consensus.cc:2229] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 3 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.737151 11825 raft_consensus.cc:2258] T 526cbf5f066347a885b5b2ef31f681d1 P aaf602cc636547d59d67849689e3c1b7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.738766 11825 tablet_replica.cc:330] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7: stopping tablet replica
I20260807 08:00:22.739190 11825 raft_consensus.cc:2229] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:22.739580 11825 raft_consensus.cc:2258] T fd1b31d21185422792582600382ca871 P aaf602cc636547d59d67849689e3c1b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.783896 11825 tablet_server.cc:194] TabletServer@127.11.140.67:0 shutdown complete.
I20260807 08:00:22.803290 11825 master.cc:537] Master@127.11.140.126:46249 shutting down...
I20260807 08:00:22.827450 11825 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:00:22.828022 11825 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:22.828362 11825 tablet_replica.cc:330] T 00000000000000000000000000000000 P e484dc6c578f4ea1acf844cb0e662be4: stopping tablet replica
I20260807 08:00:22.847064 11825 master.cc:552] Master@127.11.140.126:46249 shutdown complete.
1100218421414a9fa7abd553db3cfca7 5
166256f885304c169d14729ac6311f16 4
aaf602cc636547d59d67849689e3c1b7 1
1100218421414a9fa7abd553db3cfca7 3
166256f885304c169d14729ac6311f16 4
aaf602cc636547d59d67849689e3c1b7 3
1100218421414a9fa7abd553db3cfca7 8
166256f885304c169d14729ac6311f16 1
aaf602cc636547d59d67849689e3c1b7 1
1100218421414a9fa7abd553db3cfca7 4
166256f885304c169d14729ac6311f16 3
aaf602cc636547d59d67849689e3c1b7 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (30348 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260807 08:00:22.890411 11825 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.140.126:41033
I20260807 08:00:22.891515 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:00:22.897280 13397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:00:22.898015 13398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:00:22.899291 13401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:00:22.899513 11825 server_base.cc:1034] running on GCE node
I20260807 08:00:22.900725 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 08:00:22.900941 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 08:00:22.901099 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089622901085 us; error 0 us; skew 500 ppm
I20260807 08:00:22.901659 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 08:00:22.907608 11825 webserver.cc:460] Webserver started at http://127.11.140.126:44849/ using document root <none> and password file <none>
I20260807 08:00:22.908082 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 08:00:22.908277 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:00:22.908567 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:00:22.910059 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/master-0-root/instance:
uuid: "5d5229fe29be45b28bd5a2e9d3d998b1"
format_stamp: "Formatted at 2026-08-07 08:00:22 on dist-test-slave-zr1t"
I20260807 08:00:22.914408 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 08:00:22.917690 13411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:22.918417 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 08:00:22.918694 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/master-0-root
uuid: "5d5229fe29be45b28bd5a2e9d3d998b1"
format_stamp: "Formatted at 2026-08-07 08:00:22 on dist-test-slave-zr1t"
I20260807 08:00:22.918969 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:00:22.942878 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:00:22.944092 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:00:22.982877 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.126:41033
I20260807 08:00:22.982977 13481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.126:41033 every 8 connection(s)
I20260807 08:00:22.986626 13485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:22.995837 13485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Bootstrap starting.
I20260807 08:00:23.000069 13485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:23.004102 13485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: No bootstrap required, opened a new log
I20260807 08:00:23.006162 13485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5229fe29be45b28bd5a2e9d3d998b1" member_type: VOTER }
I20260807 08:00:23.006624 13485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:23.006793 13485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5229fe29be45b28bd5a2e9d3d998b1, State: Initialized, Role: FOLLOWER
I20260807 08:00:23.007345 13485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5229fe29be45b28bd5a2e9d3d998b1" member_type: VOTER }
I20260807 08:00:23.007772 13485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:00:23.007953 13485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:00:23.008170 13485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:23.011865 13485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5229fe29be45b28bd5a2e9d3d998b1" member_type: VOTER }
I20260807 08:00:23.012333 13485 leader_election.cc:302] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d5229fe29be45b28bd5a2e9d3d998b1; no voters:
I20260807 08:00:23.013343 13485 leader_election.cc:288] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:00:23.013761 13490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:23.015102 13490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [term 1 LEADER]: Becoming Leader. State: Replica: 5d5229fe29be45b28bd5a2e9d3d998b1, State: Running, Role: LEADER
I20260807 08:00:23.015712 13490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5229fe29be45b28bd5a2e9d3d998b1" member_type: VOTER }
I20260807 08:00:23.016247 13485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:00:23.021893 13491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d5229fe29be45b28bd5a2e9d3d998b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5229fe29be45b28bd5a2e9d3d998b1" member_type: VOTER } }
I20260807 08:00:23.022388 13493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d5229fe29be45b28bd5a2e9d3d998b1. Latest consensus state: current_term: 1 leader_uuid: "5d5229fe29be45b28bd5a2e9d3d998b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5229fe29be45b28bd5a2e9d3d998b1" member_type: VOTER } }
I20260807 08:00:23.022641 13491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [sys.catalog]: This master's current role is: LEADER
I20260807 08:00:23.022934 13493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1 [sys.catalog]: This master's current role is: LEADER
I20260807 08:00:23.025223 13499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:00:23.030129 13499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:00:23.032616 11825 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260807 08:00:23.038427 13499 catalog_manager.cc:1347] Generated new cluster ID: 65d3ca941e7c4516a6c4db0ab01d0472
I20260807 08:00:23.038658 13499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:00:23.061661 13499 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:00:23.063218 13499 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:00:23.077726 13499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Generated new TSK 0
I20260807 08:00:23.078317 13499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:00:23.099258 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:00:23.105103 13522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:00:23.106546 13523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:00:23.108114 13526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:00:23.108956 11825 server_base.cc:1034] running on GCE node
I20260807 08:00:23.110003 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 08:00:23.110177 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 08:00:23.110291 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089623110282 us; error 0 us; skew 500 ppm
I20260807 08:00:23.110737 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 08:00:23.112890 11825 webserver.cc:460] Webserver started at http://127.11.140.65:40747/ using document root <none> and password file <none>
I20260807 08:00:23.113317 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 08:00:23.113476 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:00:23.113778 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:00:23.114861 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/instance:
uuid: "77da774f2bb54cf28302d53d0db224ad"
format_stamp: "Formatted at 2026-08-07 08:00:23 on dist-test-slave-zr1t"
I20260807 08:00:23.119238 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 08:00:23.122534 13534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:23.123240 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 08:00:23.123518 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root
uuid: "77da774f2bb54cf28302d53d0db224ad"
format_stamp: "Formatted at 2026-08-07 08:00:23 on dist-test-slave-zr1t"
I20260807 08:00:23.123783 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:00:23.132481 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:00:23.133694 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:00:23.135095 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:00:23.137244 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:00:23.137462 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:23.137758 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:00:23.137934 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:23.178057 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.65:41037
I20260807 08:00:23.178156 13630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.65:41037 every 8 connection(s)
I20260807 08:00:23.182858 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:00:23.190311 13637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:00:23.190941 13638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:00:23.193758 13640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:00:23.196328 11825 server_base.cc:1034] running on GCE node
I20260807 08:00:23.198246 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 08:00:23.198500 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 08:00:23.198705 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089623198689 us; error 0 us; skew 500 ppm
I20260807 08:00:23.199424 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 08:00:23.201035 13632 heartbeater.cc:347] Connected to a master server at 127.11.140.126:41033
I20260807 08:00:23.201393 13632 heartbeater.cc:464] Registering TS with master...
I20260807 08:00:23.202150 13632 heartbeater.cc:511] Master 127.11.140.126:41033 requested a full tablet report, sending...
I20260807 08:00:23.202383 11825 webserver.cc:460] Webserver started at http://127.11.140.66:37957/ using document root <none> and password file <none>
I20260807 08:00:23.202952 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 08:00:23.203152 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:00:23.203428 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:00:23.204370 13432 ts_manager.cc:194] Registered new tserver with Master: 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:23.204957 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/instance:
uuid: "9210dbc4cc9c45e5b26a463b705d362c"
format_stamp: "Formatted at 2026-08-07 08:00:23 on dist-test-slave-zr1t"
I20260807 08:00:23.207072 13432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56582
I20260807 08:00:23.210585 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 08:00:23.213749 13652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:23.214443 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 08:00:23.214720 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root
uuid: "9210dbc4cc9c45e5b26a463b705d362c"
format_stamp: "Formatted at 2026-08-07 08:00:23 on dist-test-slave-zr1t"
I20260807 08:00:23.214977 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:00:23.238621 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:00:23.239823 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:00:23.241369 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:00:23.243498 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:00:23.243664 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:23.243831 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:00:23.243947 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:23.282497 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.66:44033
I20260807 08:00:23.282609 13743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.66:44033 every 8 connection(s)
I20260807 08:00:23.287154 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:00:23.294940 13752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:00:23.296932 13753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:00:23.299307 13758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:00:23.299746 11825 server_base.cc:1034] running on GCE node
I20260807 08:00:23.300830 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 08:00:23.301136 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 08:00:23.301343 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089623301325 us; error 0 us; skew 500 ppm
I20260807 08:00:23.302071 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 08:00:23.302696 13744 heartbeater.cc:347] Connected to a master server at 127.11.140.126:41033
I20260807 08:00:23.303148 13744 heartbeater.cc:464] Registering TS with master...
I20260807 08:00:23.304010 13744 heartbeater.cc:511] Master 127.11.140.126:41033 requested a full tablet report, sending...
I20260807 08:00:23.304875 11825 webserver.cc:460] Webserver started at http://127.11.140.67:43599/ using document root <none> and password file <none>
I20260807 08:00:23.305509 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 08:00:23.305787 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:00:23.306082 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:00:23.306340 13432 ts_manager.cc:194] Registered new tserver with Master: 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:23.307291 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/instance:
uuid: "7a889001d6494fe5b7cbbe7da1dae302"
format_stamp: "Formatted at 2026-08-07 08:00:23 on dist-test-slave-zr1t"
I20260807 08:00:23.307938 13432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56596
I20260807 08:00:23.312561 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260807 08:00:23.315876 13764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:23.316825 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 08:00:23.317137 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root
uuid: "7a889001d6494fe5b7cbbe7da1dae302"
format_stamp: "Formatted at 2026-08-07 08:00:23 on dist-test-slave-zr1t"
I20260807 08:00:23.317406 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:00:23.329772 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:00:23.330920 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:00:23.332317 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:00:23.334498 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:00:23.334697 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:23.334935 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:00:23.335086 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:23.373809 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.67:40969
I20260807 08:00:23.373904 13854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.67:40969 every 8 connection(s)
I20260807 08:00:23.388818 13855 heartbeater.cc:347] Connected to a master server at 127.11.140.126:41033
I20260807 08:00:23.389218 13855 heartbeater.cc:464] Registering TS with master...
I20260807 08:00:23.389976 13855 heartbeater.cc:511] Master 127.11.140.126:41033 requested a full tablet report, sending...
I20260807 08:00:23.391779 13432 ts_manager.cc:194] Registered new tserver with Master: 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:23.392748 11825 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015833394s
I20260807 08:00:23.393631 13432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56598
I20260807 08:00:23.395561 11825 test_util.cc:268] Using random seed: 1703550166
I20260807 08:00:23.419929 13432 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56614:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:00:23.426590 13432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:00:23.626704 13803 tablet_service.cc:1467] Processing CreateTablet for tablet 38e7d8fef17f4bdb92d0c21739339421 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260807 08:00:23.628161 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e7d8fef17f4bdb92d0c21739339421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.629846 13572 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7a6ca6100143da93759f88722f56e3 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260807 08:00:23.631247 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f7a6ca6100143da93759f88722f56e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.631034 13569 tablet_service.cc:1467] Processing CreateTablet for tablet 0248ade936924e628c46b8de6252af1b (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260807 08:00:23.632339 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0248ade936924e628c46b8de6252af1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.633291 13801 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccaeb8c899348e7959597174d89ed01 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260807 08:00:23.634625 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccaeb8c899348e7959597174d89ed01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.638696 13800 tablet_service.cc:1467] Processing CreateTablet for tablet 2acf2131e15e4fc9b2be438b9f3579b1 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260807 08:00:23.639215 13566 tablet_service.cc:1467] Processing CreateTablet for tablet c78744cb93ef47e9aacb62cbeb454457 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260807 08:00:23.640426 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2acf2131e15e4fc9b2be438b9f3579b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.641000 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78744cb93ef47e9aacb62cbeb454457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.642858 13568 tablet_service.cc:1467] Processing CreateTablet for tablet 04e678279be64e6ca8a141d6b761e522 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260807 08:00:23.644387 13568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e678279be64e6ca8a141d6b761e522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.647243 13697 tablet_service.cc:1467] Processing CreateTablet for tablet 38e7d8fef17f4bdb92d0c21739339421 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260807 08:00:23.648530 13697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e7d8fef17f4bdb92d0c21739339421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.649329 13799 tablet_service.cc:1467] Processing CreateTablet for tablet d2dfc89fb081405f96a0b53d76aeea38 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260807 08:00:23.650255 13683 tablet_service.cc:1467] Processing CreateTablet for tablet 04e678279be64e6ca8a141d6b761e522 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260807 08:00:23.650733 13696 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccaeb8c899348e7959597174d89ed01 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260807 08:00:23.651579 13683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e678279be64e6ca8a141d6b761e522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.652171 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccaeb8c899348e7959597174d89ed01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.651988 13798 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7a6ca6100143da93759f88722f56e3 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260807 08:00:23.654297 13797 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8da02e8f51481cbcca504f8cff6f80 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260807 08:00:23.654726 13694 tablet_service.cc:1467] Processing CreateTablet for tablet 2acf2131e15e4fc9b2be438b9f3579b1 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260807 08:00:23.656059 13694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2acf2131e15e4fc9b2be438b9f3579b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.656615 13795 tablet_service.cc:1467] Processing CreateTablet for tablet a01b1964575c4708bf9760dff95db7f3 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260807 08:00:23.658695 13690 tablet_service.cc:1467] Processing CreateTablet for tablet d2dfc89fb081405f96a0b53d76aeea38 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260807 08:00:23.658934 13577 tablet_service.cc:1467] Processing CreateTablet for tablet 38e7d8fef17f4bdb92d0c21739339421 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260807 08:00:23.660001 13690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2dfc89fb081405f96a0b53d76aeea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.661103 13576 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccaeb8c899348e7959597174d89ed01 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260807 08:00:23.662667 13689 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7a6ca6100143da93759f88722f56e3 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260807 08:00:23.663292 13575 tablet_service.cc:1467] Processing CreateTablet for tablet 2acf2131e15e4fc9b2be438b9f3579b1 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260807 08:00:23.663964 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f7a6ca6100143da93759f88722f56e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.665374 13573 tablet_service.cc:1467] Processing CreateTablet for tablet d2dfc89fb081405f96a0b53d76aeea38 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260807 08:00:23.666693 13682 tablet_service.cc:1467] Processing CreateTablet for tablet c78744cb93ef47e9aacb62cbeb454457 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260807 08:00:23.667547 13571 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8da02e8f51481cbcca504f8cff6f80 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260807 08:00:23.668051 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78744cb93ef47e9aacb62cbeb454457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.666706 13687 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8da02e8f51481cbcca504f8cff6f80 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260807 08:00:23.670392 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8da02e8f51481cbcca504f8cff6f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.650686 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2dfc89fb081405f96a0b53d76aeea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.673857 13795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01b1964575c4708bf9760dff95db7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.676578 13685 tablet_service.cc:1467] Processing CreateTablet for tablet 0248ade936924e628c46b8de6252af1b (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260807 08:00:23.677886 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f7a6ca6100143da93759f88722f56e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.679000 13570 tablet_service.cc:1467] Processing CreateTablet for tablet a01b1964575c4708bf9760dff95db7f3 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260807 08:00:23.681859 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8da02e8f51481cbcca504f8cff6f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.667924 13686 tablet_service.cc:1467] Processing CreateTablet for tablet a01b1964575c4708bf9760dff95db7f3 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260807 08:00:23.677923 13685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0248ade936924e628c46b8de6252af1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.690016 13686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01b1964575c4708bf9760dff95db7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.691211 13792 tablet_service.cc:1467] Processing CreateTablet for tablet 0248ade936924e628c46b8de6252af1b (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260807 08:00:23.692797 13792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0248ade936924e628c46b8de6252af1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.694835 13790 tablet_service.cc:1467] Processing CreateTablet for tablet 04e678279be64e6ca8a141d6b761e522 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260807 08:00:23.696350 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e678279be64e6ca8a141d6b761e522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.698750 13789 tablet_service.cc:1467] Processing CreateTablet for tablet c78744cb93ef47e9aacb62cbeb454457 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260807 08:00:23.700719 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78744cb93ef47e9aacb62cbeb454457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.704195 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e7d8fef17f4bdb92d0c21739339421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.705572 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2acf2131e15e4fc9b2be438b9f3579b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.709882 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccaeb8c899348e7959597174d89ed01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.729847 13571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8da02e8f51481cbcca504f8cff6f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.731463 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01b1964575c4708bf9760dff95db7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.747308 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2dfc89fb081405f96a0b53d76aeea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.756150 13790 tablet_service.cc:1467] Processing CreateTablet for tablet 35cf7298a676441c8d12e04b066ee036 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260807 08:00:23.759397 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cf7298a676441c8d12e04b066ee036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.778762 13800 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbf945943e24c5dbc8b0a0f6c2fd2fd (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260807 08:00:23.780434 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fbf945943e24c5dbc8b0a0f6c2fd2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.783090 13874 tablet_bootstrap.cc:492] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:23.789819 13683 tablet_service.cc:1467] Processing CreateTablet for tablet 25fcdcbbf735436893ce5c13fd2eb1f1 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260807 08:00:23.795094 13801 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4b1bd6a74d452e860b1420b0ef1434 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260807 08:00:23.796669 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4b1bd6a74d452e860b1420b0ef1434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.799435 13686 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbf945943e24c5dbc8b0a0f6c2fd2fd (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260807 08:00:23.800578 13799 tablet_service.cc:1467] Processing CreateTablet for tablet 25fcdcbbf735436893ce5c13fd2eb1f1 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260807 08:00:23.801962 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fcdcbbf735436893ce5c13fd2eb1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.805910 13875 tablet_bootstrap.cc:492] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:23.836494 13682 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4b1bd6a74d452e860b1420b0ef1434 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260807 08:00:23.837999 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4b1bd6a74d452e860b1420b0ef1434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.842756 13575 tablet_service.cc:1467] Processing CreateTablet for tablet 25fcdcbbf735436893ce5c13fd2eb1f1 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260807 08:00:23.844280 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fcdcbbf735436893ce5c13fd2eb1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.852876 13798 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1339f29eb34e05a3d97cef349b97c7 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260807 08:00:23.856797 13795 tablet_service.cc:1467] Processing CreateTablet for tablet d37920c01cb94122bd1c5a0bb331be31 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260807 08:00:23.858203 13795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37920c01cb94122bd1c5a0bb331be31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.863983 13572 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4b1bd6a74d452e860b1420b0ef1434 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260807 08:00:23.862488 13792 tablet_service.cc:1467] Processing CreateTablet for tablet 76bbc935aada4ceab2774e363186745d (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260807 08:00:23.865247 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4b1bd6a74d452e860b1420b0ef1434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.865248 13792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bbc935aada4ceab2774e363186745d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.865782 13686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fbf945943e24c5dbc8b0a0f6c2fd2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.865789 13875 tablet_bootstrap.cc:654] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:23.866442 13683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fcdcbbf735436893ce5c13fd2eb1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.871150 13569 tablet_service.cc:1467] Processing CreateTablet for tablet 35cf7298a676441c8d12e04b066ee036 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260807 08:00:23.874384 13790 tablet_service.cc:1467] Processing CreateTablet for tablet fa406406335b4c9688263b810ce070a9 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260807 08:00:23.885442 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa406406335b4c9688263b810ce070a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.891371 13696 tablet_service.cc:1467] Processing CreateTablet for tablet d37920c01cb94122bd1c5a0bb331be31 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260807 08:00:23.892642 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37920c01cb94122bd1c5a0bb331be31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.883991 13685 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1339f29eb34e05a3d97cef349b97c7 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260807 08:00:23.898118 13685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1339f29eb34e05a3d97cef349b97c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.902205 13876 tablet_bootstrap.cc:492] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:23.905095 13875 tablet_bootstrap.cc:492] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:23.907160 13875 ts_tablet_manager.cc:1397] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.118s user 0.009s sys 0.006s
I20260807 08:00:23.909762 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cf7298a676441c8d12e04b066ee036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.915954 13690 tablet_service.cc:1467] Processing CreateTablet for tablet 76bbc935aada4ceab2774e363186745d (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260807 08:00:23.917212 13690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bbc935aada4ceab2774e363186745d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.916924 13803 tablet_service.cc:1467] Processing CreateTablet for tablet 6962d9b636574c5e80f4e57a227e52e2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260807 08:00:23.918467 13797 tablet_service.cc:1467] Processing CreateTablet for tablet fb6a66ac7784477f904283ba3204bac6 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260807 08:00:23.919658 13694 tablet_service.cc:1467] Processing CreateTablet for tablet fa406406335b4c9688263b810ce070a9 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260807 08:00:23.922992 13575 tablet_service.cc:1467] Processing CreateTablet for tablet d6afc407b1b94edb94f8e7e8624c9221 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260807 08:00:23.879961 13697 tablet_service.cc:1467] Processing CreateTablet for tablet 35cf7298a676441c8d12e04b066ee036 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260807 08:00:23.875293 13566 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1339f29eb34e05a3d97cef349b97c7 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260807 08:00:23.930052 13874 tablet_bootstrap.cc:654] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:23.880810 13577 tablet_service.cc:1467] Processing CreateTablet for tablet d37920c01cb94122bd1c5a0bb331be31 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260807 08:00:23.911761 13568 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbf945943e24c5dbc8b0a0f6c2fd2fd (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260807 08:00:23.935727 13689 tablet_service.cc:1467] Processing CreateTablet for tablet 6962d9b636574c5e80f4e57a227e52e2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260807 08:00:23.936635 13571 tablet_service.cc:1467] Processing CreateTablet for tablet fa406406335b4c9688263b810ce070a9 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260807 08:00:23.938519 13800 tablet_service.cc:1467] Processing CreateTablet for tablet d6afc407b1b94edb94f8e7e8624c9221 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260807 08:00:23.940680 13697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cf7298a676441c8d12e04b066ee036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.910575 13573 tablet_service.cc:1467] Processing CreateTablet for tablet 76bbc935aada4ceab2774e363186745d (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260807 08:00:23.938218 13687 tablet_service.cc:1467] Processing CreateTablet for tablet fb6a66ac7784477f904283ba3204bac6 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260807 08:00:23.942806 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6a66ac7784477f904283ba3204bac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.947670 13876 tablet_bootstrap.cc:654] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:23.952857 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6962d9b636574c5e80f4e57a227e52e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.953771 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1339f29eb34e05a3d97cef349b97c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.956703 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6afc407b1b94edb94f8e7e8624c9221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.957883 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6962d9b636574c5e80f4e57a227e52e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.928969 13801 tablet_service.cc:1467] Processing CreateTablet for tablet cc32413495ee41909c7c5494c8095084 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260807 08:00:23.960588 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6afc407b1b94edb94f8e7e8624c9221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.966382 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1339f29eb34e05a3d97cef349b97c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.969893 13568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fbf945943e24c5dbc8b0a0f6c2fd2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.962703 13682 tablet_service.cc:1467] Processing CreateTablet for tablet cc32413495ee41909c7c5494c8095084 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260807 08:00:23.971935 13571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa406406335b4c9688263b810ce070a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.972618 13570 tablet_service.cc:1467] Processing CreateTablet for tablet fb6a66ac7784477f904283ba3204bac6 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260807 08:00:23.973820 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bbc935aada4ceab2774e363186745d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.977708 13694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa406406335b4c9688263b810ce070a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.977818 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37920c01cb94122bd1c5a0bb331be31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.978583 13696 tablet_service.cc:1467] Processing CreateTablet for tablet d6afc407b1b94edb94f8e7e8624c9221 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260807 08:00:23.979815 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6afc407b1b94edb94f8e7e8624c9221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.981798 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc32413495ee41909c7c5494c8095084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.982318 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6a66ac7784477f904283ba3204bac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:23.987581 13875 raft_consensus.cc:348] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:23.988348 13875 raft_consensus.cc:374] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:23.988633 13875 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:23.989390 13875 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:23.959661 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6a66ac7784477f904283ba3204bac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.008612 13790 tablet_service.cc:1467] Processing CreateTablet for tablet 88e3dc2093eb4153a975824017d2aafb (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260807 08:00:24.008996 13795 tablet_service.cc:1467] Processing CreateTablet for tablet 0e72e5415e504959b4763e580400f8ee (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260807 08:00:24.022861 13569 tablet_service.cc:1467] Processing CreateTablet for tablet cc32413495ee41909c7c5494c8095084 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260807 08:00:24.006115 13792 tablet_service.cc:1467] Processing CreateTablet for tablet 5c683c4293d64060bd1a28772f77225a (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260807 08:00:24.024494 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc32413495ee41909c7c5494c8095084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.025131 13792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c683c4293d64060bd1a28772f77225a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.025748 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e3dc2093eb4153a975824017d2aafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.028599 13697 tablet_service.cc:1467] Processing CreateTablet for tablet 5c683c4293d64060bd1a28772f77225a (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
W20260807 08:00:24.029081 13508 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 08:00:24.030263 13795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e72e5415e504959b4763e580400f8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.037952 13874 tablet_bootstrap.cc:492] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:24.038435 13874 ts_tablet_manager.cc:1397] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.256s user 0.006s sys 0.013s
I20260807 08:00:24.041129 13874 raft_consensus.cc:348] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.041883 13874 raft_consensus.cc:374] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:24.042152 13874 raft_consensus.cc:729] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:24.043146 13697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c683c4293d64060bd1a28772f77225a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.042977 13874 consensus_queue.cc:260] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.053638 13799 tablet_service.cc:1467] Processing CreateTablet for tablet d44a80bdfaf145389eacef429f26eedc (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260807 08:00:24.056216 13689 tablet_service.cc:1467] Processing CreateTablet for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260807 08:00:24.057641 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.058562 13690 tablet_service.cc:1467] Processing CreateTablet for tablet 88e3dc2093eb4153a975824017d2aafb (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260807 08:00:24.059813 13690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e3dc2093eb4153a975824017d2aafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.071794 13875 ts_tablet_manager.cc:1428] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.164s user 0.012s sys 0.017s
I20260807 08:00:24.072585 13744 heartbeater.cc:503] Master 127.11.140.126:41033 was elected leader, sending a full tablet report...
I20260807 08:00:24.077536 13876 tablet_bootstrap.cc:492] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:24.078022 13876 ts_tablet_manager.cc:1397] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.176s user 0.004s sys 0.023s
I20260807 08:00:23.962728 13576 tablet_service.cc:1467] Processing CreateTablet for tablet 6962d9b636574c5e80f4e57a227e52e2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260807 08:00:24.080695 13876 raft_consensus.cc:348] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.088306 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44a80bdfaf145389eacef429f26eedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.081684 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc32413495ee41909c7c5494c8095084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.082969 13566 tablet_service.cc:1467] Processing CreateTablet for tablet 5c683c4293d64060bd1a28772f77225a (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260807 08:00:24.081440 13876 raft_consensus.cc:374] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:24.093324 13789 tablet_service.cc:1467] Processing CreateTablet for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260807 08:00:24.102164 13874 ts_tablet_manager.cc:1428] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.063s user 0.012s sys 0.001s
I20260807 08:00:24.082379 13683 tablet_service.cc:1467] Processing CreateTablet for tablet 0e72e5415e504959b4763e580400f8ee (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260807 08:00:24.103688 13683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e72e5415e504959b4763e580400f8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.105892 13685 tablet_service.cc:1467] Processing CreateTablet for tablet 069b769373e144929c9253bf3fdd0ab4 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260807 08:00:24.106155 13572 tablet_service.cc:1467] Processing CreateTablet for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260807 08:00:24.107134 13685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069b769373e144929c9253bf3fdd0ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.107296 13697 tablet_service.cc:1467] Processing CreateTablet for tablet 23f6b5b24a8345d0ba918651b4a2c85f (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260807 08:00:24.108457 13697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f6b5b24a8345d0ba918651b4a2c85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.110070 13855 heartbeater.cc:503] Master 127.11.140.126:41033 was elected leader, sending a full tablet report...
I20260807 08:00:24.112901 13874 tablet_bootstrap.cc:492] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:24.114632 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.095309 13876 raft_consensus.cc:729] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:24.124090 13875 tablet_bootstrap.cc:492] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:24.128688 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6962d9b636574c5e80f4e57a227e52e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.094208 13686 tablet_service.cc:1467] Processing CreateTablet for tablet d44a80bdfaf145389eacef429f26eedc (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260807 08:00:24.142341 13876 consensus_queue.cc:260] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.150664 13686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44a80bdfaf145389eacef429f26eedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.156903 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.147929 13687 tablet_service.cc:1467] Processing CreateTablet for tablet b74446812e814c7e9b4a1bd80bfb485d (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260807 08:00:24.162767 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74446812e814c7e9b4a1bd80bfb485d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.165477 13875 tablet_bootstrap.cc:654] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:24.166774 13568 tablet_service.cc:1467] Processing CreateTablet for tablet 88e3dc2093eb4153a975824017d2aafb (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260807 08:00:24.165928 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c683c4293d64060bd1a28772f77225a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.170512 13571 tablet_service.cc:1467] Processing CreateTablet for tablet 0e72e5415e504959b4763e580400f8ee (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260807 08:00:24.171742 13571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e72e5415e504959b4763e580400f8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.173943 13573 tablet_service.cc:1467] Processing CreateTablet for tablet d44a80bdfaf145389eacef429f26eedc (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260807 08:00:24.175341 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44a80bdfaf145389eacef429f26eedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.178968 13696 tablet_service.cc:1467] Processing CreateTablet for tablet 00c1526fe2ae48f3a1bcce761ff91f72 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260807 08:00:24.180253 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c1526fe2ae48f3a1bcce761ff91f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.185022 13798 tablet_service.cc:1467] Processing CreateTablet for tablet 00c1526fe2ae48f3a1bcce761ff91f72 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260807 08:00:24.186354 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c1526fe2ae48f3a1bcce761ff91f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.186411 13801 tablet_service.cc:1467] Processing CreateTablet for tablet 069b769373e144929c9253bf3fdd0ab4 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260807 08:00:24.193645 13792 tablet_service.cc:1467] Processing CreateTablet for tablet b74446812e814c7e9b4a1bd80bfb485d (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260807 08:00:24.199532 13797 tablet_service.cc:1467] Processing CreateTablet for tablet 23f6b5b24a8345d0ba918651b4a2c85f (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260807 08:00:24.197170 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069b769373e144929c9253bf3fdd0ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.206624 13792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74446812e814c7e9b4a1bd80bfb485d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.206918 13568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e3dc2093eb4153a975824017d2aafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.210778 13632 heartbeater.cc:503] Master 127.11.140.126:41033 was elected leader, sending a full tablet report...
I20260807 08:00:24.211144 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f6b5b24a8345d0ba918651b4a2c85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.219287 13575 tablet_service.cc:1467] Processing CreateTablet for tablet 00c1526fe2ae48f3a1bcce761ff91f72 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260807 08:00:24.250082 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c1526fe2ae48f3a1bcce761ff91f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.227741 13803 tablet_service.cc:1467] Processing CreateTablet for tablet 966e99d2078c4b1c9fc921c1666b45a2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260807 08:00:24.255168 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966e99d2078c4b1c9fc921c1666b45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.229316 13884 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:24.257228 13884 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.274202 13876 ts_tablet_manager.cc:1428] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.196s user 0.023s sys 0.018s
I20260807 08:00:24.275185 13876 tablet_bootstrap.cc:492] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
W20260807 08:00:24.281968 13415 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 99044.7us
I20260807 08:00:24.289034 13577 tablet_service.cc:1467] Processing CreateTablet for tablet 069b769373e144929c9253bf3fdd0ab4 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260807 08:00:24.290724 13875 tablet_bootstrap.cc:492] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:24.298961 13884 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:00:24.303052 13418 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 75926us
I20260807 08:00:24.307575 13875 ts_tablet_manager.cc:1397] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.213s user 0.016s sys 0.007s
I20260807 08:00:24.319227 13875 raft_consensus.cc:348] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.320116 13875 raft_consensus.cc:374] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:24.321842 13875 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:24.322593 13875 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.327523 13875 ts_tablet_manager.cc:1428] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20260807 08:00:24.328475 13875 tablet_bootstrap.cc:492] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:24.329102 13694 tablet_service.cc:1467] Processing CreateTablet for tablet 966e99d2078c4b1c9fc921c1666b45a2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260807 08:00:24.346383 13875 tablet_bootstrap.cc:654] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:24.354997 13570 tablet_service.cc:1467] Processing CreateTablet for tablet 23f6b5b24a8345d0ba918651b4a2c85f (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260807 08:00:24.367139 13569 tablet_service.cc:1467] Processing CreateTablet for tablet b74446812e814c7e9b4a1bd80bfb485d (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260807 08:00:24.379730 13882 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:24.380241 13882 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.377960 13694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966e99d2078c4b1c9fc921c1666b45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.383883 13882 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:24.384366 13876 tablet_bootstrap.cc:654] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:24.386828 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069b769373e144929c9253bf3fdd0ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.400413 13687 tablet_service.cc:1467] Processing CreateTablet for tablet 2de183214b52498faef7f8a8f74a2adc (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260807 08:00:24.401726 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de183214b52498faef7f8a8f74a2adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.402153 13685 tablet_service.cc:1467] Processing CreateTablet for tablet e4e01d60f0064bb3958a65250e430990 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260807 08:00:24.403354 13685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e01d60f0064bb3958a65250e430990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.389117 13874 tablet_bootstrap.cc:654] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:24.403059 13789 tablet_service.cc:1467] Processing CreateTablet for tablet 2de183214b52498faef7f8a8f74a2adc (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260807 08:00:24.414462 13682 tablet_service.cc:1467] Processing CreateTablet for tablet b1c0fb87eb274da9bfb29c3b9300dc1c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260807 08:00:24.426937 13576 tablet_service.cc:1467] Processing CreateTablet for tablet 966e99d2078c4b1c9fc921c1666b45a2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20260807 08:00:24.437844 13633 tablet.cc:2369] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:00:24.390342 13690 tablet_service.cc:1467] Processing CreateTablet for tablet 1a55b0fbe6eb48649fd7cfc0be18010d (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260807 08:00:24.389716 13790 tablet_service.cc:1467] Processing CreateTablet for tablet 1a55b0fbe6eb48649fd7cfc0be18010d (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260807 08:00:24.444715 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74446812e814c7e9b4a1bd80bfb485d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.447997 13882 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:24.455108 13882 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.457182 13882 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:24.468405 13883 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:24.476985 13883 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.479396 13883 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:24.472963 13690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a55b0fbe6eb48649fd7cfc0be18010d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.476022 13683 tablet_service.cc:1467] Processing CreateTablet for tablet d324e0b3eeee43549f0b430264c5d25b (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260807 08:00:24.494458 13572 tablet_service.cc:1467] Processing CreateTablet for tablet e4e01d60f0064bb3958a65250e430990 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260807 08:00:24.496974 13566 tablet_service.cc:1467] Processing CreateTablet for tablet b1c0fb87eb274da9bfb29c3b9300dc1c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260807 08:00:24.484915 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f6b5b24a8345d0ba918651b4a2c85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.498307 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e01d60f0064bb3958a65250e430990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.499423 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de183214b52498faef7f8a8f74a2adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.513031 13571 tablet_service.cc:1467] Processing CreateTablet for tablet 2de183214b52498faef7f8a8f74a2adc (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260807 08:00:24.522506 13689 tablet_service.cc:1467] Processing CreateTablet for tablet 4c9ed355e84f4fc7a80a1d59a76a365c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260807 08:00:24.528424 13797 tablet_service.cc:1467] Processing CreateTablet for tablet e4e01d60f0064bb3958a65250e430990 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260807 08:00:24.531616 13800 tablet_service.cc:1467] Processing CreateTablet for tablet b1c0fb87eb274da9bfb29c3b9300dc1c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260807 08:00:24.544019 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c0fb87eb274da9bfb29c3b9300dc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.543143 13683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d324e0b3eeee43549f0b430264c5d25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.564409 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:24.566032 13537 leader_election.cc:341] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:24.567952 13795 tablet_service.cc:1467] Processing CreateTablet for tablet d324e0b3eeee43549f0b430264c5d25b (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260807 08:00:24.555171 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a55b0fbe6eb48649fd7cfc0be18010d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.569769 13875 tablet_bootstrap.cc:492] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:24.570233 13875 ts_tablet_manager.cc:1397] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.242s user 0.014s sys 0.023s
I20260807 08:00:24.576851 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c9ed355e84f4fc7a80a1d59a76a365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.588812 13573 tablet_service.cc:1467] Processing CreateTablet for tablet 1a55b0fbe6eb48649fd7cfc0be18010d (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260807 08:00:24.589705 13801 tablet_service.cc:1467] Processing CreateTablet for tablet b122c69d5e2045e380bdc9f6f8b487d2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260807 08:00:24.590169 13803 tablet_service.cc:1467] Processing CreateTablet for tablet 4c9ed355e84f4fc7a80a1d59a76a365c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260807 08:00:24.592082 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c0fb87eb274da9bfb29c3b9300dc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.592912 13875 raft_consensus.cc:348] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.593726 13875 raft_consensus.cc:374] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:24.594645 13875 raft_consensus.cc:729] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:24.583460 13571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de183214b52498faef7f8a8f74a2adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.602653 13798 tablet_service.cc:1467] Processing CreateTablet for tablet 2e37d72ff9b640289fad70dde36572fe (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260807 08:00:24.603735 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a55b0fbe6eb48649fd7cfc0be18010d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.605423 13792 tablet_service.cc:1467] Processing CreateTablet for tablet d5c39633297442a8a7c63b17b17cbe54 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260807 08:00:24.561148 13568 tablet_service.cc:1467] Processing CreateTablet for tablet d324e0b3eeee43549f0b430264c5d25b (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260807 08:00:24.617566 13568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d324e0b3eeee43549f0b430264c5d25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.619270 13876 tablet_bootstrap.cc:492] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:24.623867 13696 tablet_service.cc:1467] Processing CreateTablet for tablet b122c69d5e2045e380bdc9f6f8b487d2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260807 08:00:24.586800 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e01d60f0064bb3958a65250e430990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.633384 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966e99d2078c4b1c9fc921c1666b45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.635433 13686 tablet_service.cc:1467] Processing CreateTablet for tablet 2e37d72ff9b640289fad70dde36572fe (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260807 08:00:24.636690 13686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e37d72ff9b640289fad70dde36572fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.613415 13799 tablet_service.cc:1467] Processing CreateTablet for tablet 4614fbd9fc524f61bb9d11ea324e783e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260807 08:00:24.623658 13875 consensus_queue.cc:260] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.640976 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:24.642525 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:24.651218 13575 tablet_service.cc:1467] Processing CreateTablet for tablet 4c9ed355e84f4fc7a80a1d59a76a365c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260807 08:00:24.652554 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c9ed355e84f4fc7a80a1d59a76a365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.653970 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4614fbd9fc524f61bb9d11ea324e783e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 08:00:24.655426 13661 leader_election.cc:341] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:24.656072 13661 leader_election.cc:341] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:24.657918 13792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c39633297442a8a7c63b17b17cbe54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.659972 13690 tablet_service.cc:1467] Processing CreateTablet for tablet d5c39633297442a8a7c63b17b17cbe54 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260807 08:00:24.661198 13690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c39633297442a8a7c63b17b17cbe54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.662101 13795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d324e0b3eeee43549f0b430264c5d25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.663749 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b122c69d5e2045e380bdc9f6f8b487d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.665648 13876 ts_tablet_manager.cc:1397] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.391s user 0.011s sys 0.011s
I20260807 08:00:24.666119 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c9ed355e84f4fc7a80a1d59a76a365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.668417 13876 raft_consensus.cc:348] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.669193 13876 raft_consensus.cc:374] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:24.669795 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c0fb87eb274da9bfb29c3b9300dc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.674710 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:00:24.676374 13540 leader_election.cc:341] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:24.676796 13540 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:24.677083 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b122c69d5e2045e380bdc9f6f8b487d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.676889 13685 tablet_service.cc:1467] Processing CreateTablet for tablet 4614fbd9fc524f61bb9d11ea324e783e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260807 08:00:24.682341 13685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4614fbd9fc524f61bb9d11ea324e783e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.686340 13884 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:24.696203 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:24.702198 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:24.702250 13769 leader_election.cc:341] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:24.703737 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:24.704288 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:00:24.705650 13657 leader_election.cc:341] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:24.706085 13657 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:24.706868 13882 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 08:00:24.707857 13657 leader_election.cc:341] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:24.708268 13657 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:24.708958 13882 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 08:00:24.710346 13769 leader_election.cc:341] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:24.710760 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:24.711493 13883 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:24.713816 13874 tablet_bootstrap.cc:492] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:24.714278 13874 ts_tablet_manager.cc:1397] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.602s user 0.026s sys 0.011s
I20260807 08:00:24.717248 13789 tablet_service.cc:1467] Processing CreateTablet for tablet e110e8648fd04684ae2bf4d0c71752a3 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260807 08:00:24.718526 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e110e8648fd04684ae2bf4d0c71752a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.717120 13874 raft_consensus.cc:348] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.719934 13874 raft_consensus.cc:374] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:24.720235 13874 raft_consensus.cc:729] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:24.724740 13874 consensus_queue.cc:260] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.729750 13874 ts_tablet_manager.cc:1428] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260807 08:00:24.730654 13874 tablet_bootstrap.cc:492] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:24.669483 13876 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:24.734118 13876 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.736493 13876 ts_tablet_manager.cc:1428] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.070s user 0.005s sys 0.001s
I20260807 08:00:24.737423 13876 tablet_bootstrap.cc:492] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:24.742638 13875 ts_tablet_manager.cc:1428] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.172s user 0.001s sys 0.006s
I20260807 08:00:24.743659 13875 tablet_bootstrap.cc:492] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:24.746379 13697 tablet_service.cc:1467] Processing CreateTablet for tablet e110e8648fd04684ae2bf4d0c71752a3 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260807 08:00:24.747612 13697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e110e8648fd04684ae2bf4d0c71752a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.716416 13790 tablet_service.cc:1467] Processing CreateTablet for tablet 74ccb94230614cc48e846580ad82e994 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260807 08:00:24.751181 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ccb94230614cc48e846580ad82e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.753935 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e37d72ff9b640289fad70dde36572fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.767256 13577 tablet_service.cc:1467] Processing CreateTablet for tablet b122c69d5e2045e380bdc9f6f8b487d2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260807 08:00:24.768779 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b122c69d5e2045e380bdc9f6f8b487d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.769173 13874 tablet_bootstrap.cc:654] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:24.774180 13694 tablet_service.cc:1467] Processing CreateTablet for tablet 74ccb94230614cc48e846580ad82e994 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260807 08:00:24.790699 13687 tablet_service.cc:1467] Processing CreateTablet for tablet 13ee9b6f7fa547a3badcb0eba0210119 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260807 08:00:24.811044 13569 tablet_service.cc:1467] Processing CreateTablet for tablet d5c39633297442a8a7c63b17b17cbe54 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260807 08:00:24.812369 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c39633297442a8a7c63b17b17cbe54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.812068 13789 tablet_service.cc:1467] Processing CreateTablet for tablet 13ee9b6f7fa547a3badcb0eba0210119 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260807 08:00:24.813257 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ee9b6f7fa547a3badcb0eba0210119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.813740 13576 tablet_service.cc:1467] Processing CreateTablet for tablet 4614fbd9fc524f61bb9d11ea324e783e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260807 08:00:24.816582 13694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ccb94230614cc48e846580ad82e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.817907 13566 tablet_service.cc:1467] Processing CreateTablet for tablet 13ee9b6f7fa547a3badcb0eba0210119 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260807 08:00:24.822547 13874 tablet_bootstrap.cc:492] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:24.822389 13683 tablet_service.cc:1467] Processing CreateTablet for tablet 1588644cd4f943a7b5d776326af48218 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260807 08:00:24.823025 13874 ts_tablet_manager.cc:1397] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.030s
I20260807 08:00:24.823100 13571 tablet_service.cc:1467] Processing CreateTablet for tablet 74ccb94230614cc48e846580ad82e994 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260807 08:00:24.823633 13683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1588644cd4f943a7b5d776326af48218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.828997 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ee9b6f7fa547a3badcb0eba0210119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.830375 13682 tablet_service.cc:1467] Processing CreateTablet for tablet b17c7c715cf146728d57366f8e070d31 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260807 08:00:24.831578 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17c7c715cf146728d57366f8e070d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.835155 13876 tablet_bootstrap.cc:654] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:24.827869 13799 tablet_service.cc:1467] Processing CreateTablet for tablet b17c7c715cf146728d57366f8e070d31 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260807 08:00:24.810562 13570 tablet_service.cc:1467] Processing CreateTablet for tablet 2e37d72ff9b640289fad70dde36572fe (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260807 08:00:24.836241 13874 raft_consensus.cc:348] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.851991 13874 raft_consensus.cc:374] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:24.852314 13874 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:24.852869 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17c7c715cf146728d57366f8e070d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.852686 13789 tablet_service.cc:1467] Processing CreateTablet for tablet 70356515c14242539aa745f7e028f6b6 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260807 08:00:24.853045 13874 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.854161 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70356515c14242539aa745f7e028f6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.855284 13874 ts_tablet_manager.cc:1428] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.032s user 0.007s sys 0.001s
I20260807 08:00:24.838316 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4614fbd9fc524f61bb9d11ea324e783e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.841564 13689 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab8c56441f84d86b5798c528463965c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260807 08:00:24.856726 13571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ccb94230614cc48e846580ad82e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.856856 13572 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab8c56441f84d86b5798c528463965c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260807 08:00:24.857916 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ee9b6f7fa547a3badcb0eba0210119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.860378 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab8c56441f84d86b5798c528463965c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.860486 13795 tablet_service.cc:1467] Processing CreateTablet for tablet 232127da9f574f41a5a8087ae775ef6b (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260807 08:00:24.862139 13874 tablet_bootstrap.cc:492] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:24.866535 13800 tablet_service.cc:1467] Processing CreateTablet for tablet e67bd63471c94c19a092c8f1aef64408 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260807 08:00:24.871107 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e37d72ff9b640289fad70dde36572fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.874526 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab8c56441f84d86b5798c528463965c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.875367 13883 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:24.875941 13883 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.879113 13883 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:24.881333 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:24.882010 13591 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:24.883072 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:24.883745 13711 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:24.885305 13769 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:24.886596 13883 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:24.886958 13883 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:24.887277 13883 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:24.845508 13686 tablet_service.cc:1467] Processing CreateTablet for tablet c8727dbda122418487cb692370c2399e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260807 08:00:24.894001 13795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232127da9f574f41a5a8087ae775ef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.839860 13803 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab8c56441f84d86b5798c528463965c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260807 08:00:24.896900 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67bd63471c94c19a092c8f1aef64408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.816586 13573 tablet_service.cc:1467] Processing CreateTablet for tablet e110e8648fd04684ae2bf4d0c71752a3 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260807 08:00:24.897960 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e110e8648fd04684ae2bf4d0c71752a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.900589 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab8c56441f84d86b5798c528463965c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.854210 13875 tablet_bootstrap.cc:654] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:24.835283 13792 tablet_service.cc:1467] Processing CreateTablet for tablet 1588644cd4f943a7b5d776326af48218 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260807 08:00:24.907893 13792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1588644cd4f943a7b5d776326af48218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.911106 13874 tablet_bootstrap.cc:654] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:24.846280 13575 tablet_service.cc:1467] Processing CreateTablet for tablet 1588644cd4f943a7b5d776326af48218 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260807 08:00:24.914270 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1588644cd4f943a7b5d776326af48218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.849349 13797 tablet_service.cc:1467] Processing CreateTablet for tablet c8727dbda122418487cb692370c2399e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260807 08:00:24.917732 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8727dbda122418487cb692370c2399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.932376 13801 tablet_service.cc:1467] Processing CreateTablet for tablet c02eb1166e17491f82cfbd9c5cd030a2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260807 08:00:24.933759 13568 tablet_service.cc:1467] Processing CreateTablet for tablet b17c7c715cf146728d57366f8e070d31 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260807 08:00:24.935010 13568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17c7c715cf146728d57366f8e070d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.952208 13686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8727dbda122418487cb692370c2399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.954285 13697 tablet_service.cc:1467] Processing CreateTablet for tablet 70356515c14242539aa745f7e028f6b6 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260807 08:00:24.955561 13697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70356515c14242539aa745f7e028f6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.957970 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02eb1166e17491f82cfbd9c5cd030a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.961020 13696 tablet_service.cc:1467] Processing CreateTablet for tablet 232127da9f574f41a5a8087ae775ef6b (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260807 08:00:24.962468 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232127da9f574f41a5a8087ae775ef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.961117 13687 tablet_service.cc:1467] Processing CreateTablet for tablet e67bd63471c94c19a092c8f1aef64408 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260807 08:00:24.944753 13883 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.967620 13883 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:24.967705 13685 tablet_service.cc:1467] Processing CreateTablet for tablet 466eca488bfd44d396c22e2396d1ef7e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260807 08:00:24.986021 13685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 466eca488bfd44d396c22e2396d1ef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.986685 13573 tablet_service.cc:1467] Processing CreateTablet for tablet c02eb1166e17491f82cfbd9c5cd030a2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260807 08:00:24.987738 13690 tablet_service.cc:1467] Processing CreateTablet for tablet e5ef275b7c0a45619cbc05266aea24dc (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260807 08:00:24.988670 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02eb1166e17491f82cfbd9c5cd030a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.988956 13690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ef275b7c0a45619cbc05266aea24dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.977948 13883 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:24.993814 13883 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:24.995743 13883 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:25.000458 13798 tablet_service.cc:1467] Processing CreateTablet for tablet e5ef275b7c0a45619cbc05266aea24dc (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260807 08:00:25.001830 13572 tablet_service.cc:1467] Processing CreateTablet for tablet 466eca488bfd44d396c22e2396d1ef7e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260807 08:00:25.003134 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 466eca488bfd44d396c22e2396d1ef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.006034 13577 tablet_service.cc:1467] Processing CreateTablet for tablet 232127da9f574f41a5a8087ae775ef6b (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260807 08:00:25.007190 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232127da9f574f41a5a8087ae775ef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.009357 13571 tablet_service.cc:1467] Processing CreateTablet for tablet e5ef275b7c0a45619cbc05266aea24dc (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260807 08:00:25.010576 13571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ef275b7c0a45619cbc05266aea24dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.010265 13576 tablet_service.cc:1467] Processing CreateTablet for tablet e67bd63471c94c19a092c8f1aef64408 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260807 08:00:25.011474 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67bd63471c94c19a092c8f1aef64408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.013027 13689 tablet_service.cc:1467] Processing CreateTablet for tablet 6f5ccbaa50ae4451a35038284693254c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260807 08:00:25.014210 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f5ccbaa50ae4451a35038284693254c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:24.965881 13569 tablet_service.cc:1467] Processing CreateTablet for tablet c8727dbda122418487cb692370c2399e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260807 08:00:24.967708 13683 tablet_service.cc:1467] Processing CreateTablet for tablet c02eb1166e17491f82cfbd9c5cd030a2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260807 08:00:25.025790 13882 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.026294 13882 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.027822 13566 tablet_service.cc:1467] Processing CreateTablet for tablet 6f5ccbaa50ae4451a35038284693254c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260807 08:00:25.028225 13882 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.030579 13682 tablet_service.cc:1467] Processing CreateTablet for tablet b09437796639470caac9548c71d72fb2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
W20260807 08:00:25.031364 13508 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 08:00:25.032733 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:25.034166 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67bd63471c94c19a092c8f1aef64408. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 08:00:25.034699 13657 leader_election.cc:341] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.036166 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:00:25.037602 13661 leader_election.cc:341] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.038033 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:25.039148 13882 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:25.045089 13573 tablet_service.cc:1467] Processing CreateTablet for tablet b09437796639470caac9548c71d72fb2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260807 08:00:25.046044 13570 tablet_service.cc:1467] Processing CreateTablet for tablet 70356515c14242539aa745f7e028f6b6 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260807 08:00:25.046044 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ef275b7c0a45619cbc05266aea24dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.051030 13876 tablet_bootstrap.cc:492] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:25.051504 13876 ts_tablet_manager.cc:1397] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.314s user 0.011s sys 0.006s
I20260807 08:00:25.063874 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8727dbda122418487cb692370c2399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.062320 13683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02eb1166e17491f82cfbd9c5cd030a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.069829 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70356515c14242539aa745f7e028f6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.069660 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:25.070312 13591 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.071923 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09437796639470caac9548c71d72fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.077235 13694 tablet_service.cc:1467] Processing CreateTablet for tablet caa5c12b1b5e4feea56543c706f21ec2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260807 08:00:25.077966 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f5ccbaa50ae4451a35038284693254c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.078506 13694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa5c12b1b5e4feea56543c706f21ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.081913 13876 raft_consensus.cc:348] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.082679 13876 raft_consensus.cc:374] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.082980 13876 raft_consensus.cc:729] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.083735 13876 consensus_queue.cc:260] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.092974 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09437796639470caac9548c71d72fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.094074 13799 tablet_service.cc:1467] Processing CreateTablet for tablet 466eca488bfd44d396c22e2396d1ef7e (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260807 08:00:25.095366 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 466eca488bfd44d396c22e2396d1ef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.104262 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:25.104867 13711 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.105221 13875 tablet_bootstrap.cc:492] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:25.105742 13875 ts_tablet_manager.cc:1397] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.362s user 0.020s sys 0.010s
I20260807 08:00:25.109613 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:25.110263 13592 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:25.110375 13875 raft_consensus.cc:348] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.120983 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:25.121244 13875 raft_consensus.cc:374] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.122799 13875 raft_consensus.cc:729] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.121021 13689 tablet_service.cc:1467] Processing CreateTablet for tablet 06a735af95cd49e1942d669d2e5d91c6 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260807 08:00:25.127619 13876 ts_tablet_manager.cc:1428] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.076s user 0.009s sys 0.000s
I20260807 08:00:25.128535 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a735af95cd49e1942d669d2e5d91c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.130264 13711 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:25.139710 13591 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:25.153424 13876 tablet_bootstrap.cc:492] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:25.153978 13577 tablet_service.cc:1467] Processing CreateTablet for tablet 06a735af95cd49e1942d669d2e5d91c6 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260807 08:00:25.155275 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a735af95cd49e1942d669d2e5d91c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.143767 13875 consensus_queue.cc:260] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.159245 13876 tablet_bootstrap.cc:654] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.159627 13875 ts_tablet_manager.cc:1428] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.054s user 0.006s sys 0.000s
I20260807 08:00:25.178162 13803 tablet_service.cc:1467] Processing CreateTablet for tablet 6f5ccbaa50ae4451a35038284693254c (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260807 08:00:25.178468 13801 tablet_service.cc:1467] Processing CreateTablet for tablet b09437796639470caac9548c71d72fb2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260807 08:00:25.179477 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f5ccbaa50ae4451a35038284693254c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.180073 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09437796639470caac9548c71d72fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.191056 13795 tablet_service.cc:1467] Processing CreateTablet for tablet 06a735af95cd49e1942d669d2e5d91c6 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260807 08:00:25.192268 13795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a735af95cd49e1942d669d2e5d91c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.192440 13798 tablet_service.cc:1467] Processing CreateTablet for tablet caa5c12b1b5e4feea56543c706f21ec2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260807 08:00:25.193636 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa5c12b1b5e4feea56543c706f21ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.198184 13875 tablet_bootstrap.cc:492] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
W20260807 08:00:25.214435 13769 leader_election.cc:341] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.215154 13769 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:25.216147 13912 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.216675 13912 raft_consensus.cc:686] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:25.174492 13575 tablet_service.cc:1467] Processing CreateTablet for tablet caa5c12b1b5e4feea56543c706f21ec2 (DEFAULT_TABLE table=test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260807 08:00:25.217911 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa5c12b1b5e4feea56543c706f21ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:25.217490 13912 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.225651 13874 tablet_bootstrap.cc:492] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:25.226179 13874 ts_tablet_manager.cc:1397] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.364s user 0.035s sys 0.045s
I20260807 08:00:25.216158 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:25.228783 13883 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:25.229156 13883 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:25.229511 13883 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.228909 13874 raft_consensus.cc:348] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.240401 13883 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.242939 13883 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:25.243067 13875 tablet_bootstrap.cc:654] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.244676 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:25.244951 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:25.245298 13591 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.236737 13874 raft_consensus.cc:374] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260807 08:00:25.246186 13769 leader_election.cc:341] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.246511 13874 raft_consensus.cc:729] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.248594 13876 tablet_bootstrap.cc:492] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:25.250487 13876 ts_tablet_manager.cc:1397] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.098s user 0.010s sys 0.014s
I20260807 08:00:25.253553 13876 raft_consensus.cc:348] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.254282 13876 raft_consensus.cc:374] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.254561 13876 raft_consensus.cc:729] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.255378 13876 consensus_queue.cc:260] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.256580 13591 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:25.258361 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:25.259261 13912 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.259737 13912 raft_consensus.cc:686] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:25.260092 13875 tablet_bootstrap.cc:492] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:25.260542 13875 ts_tablet_manager.cc:1397] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.084s user 0.012s sys 0.003s
I20260807 08:00:25.260572 13912 consensus_queue.cc:237] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.261260 13428 catalog_manager.cc:5495] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:25.263404 13875 raft_consensus.cc:348] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } }
I20260807 08:00:25.264102 13875 raft_consensus.cc:374] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.264401 13875 raft_consensus.cc:729] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.265161 13875 consensus_queue.cc:260] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } }
I20260807 08:00:25.266137 13876 ts_tablet_manager.cc:1428] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260807 08:00:25.266984 13874 consensus_queue.cc:260] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.270924 13875 ts_tablet_manager.cc:1428] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260807 08:00:25.271889 13875 tablet_bootstrap.cc:492] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:25.277819 13875 tablet_bootstrap.cc:654] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.288245 13876 tablet_bootstrap.cc:492] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:25.289160 13875 tablet_bootstrap.cc:492] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:25.289661 13875 ts_tablet_manager.cc:1397] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20260807 08:00:25.292286 13875 raft_consensus.cc:348] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.293045 13875 raft_consensus.cc:374] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.293345 13875 raft_consensus.cc:729] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.294298 13875 consensus_queue.cc:260] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.295526 13874 ts_tablet_manager.cc:1428] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.069s user 0.005s sys 0.001s
I20260807 08:00:25.296535 13874 tablet_bootstrap.cc:492] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:25.302556 13876 tablet_bootstrap.cc:654] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.304165 13875 ts_tablet_manager.cc:1428] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260807 08:00:25.306497 13874 tablet_bootstrap.cc:654] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.310330 13875 tablet_bootstrap.cc:492] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:25.314790 13874 tablet_bootstrap.cc:492] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:25.314790 13876 tablet_bootstrap.cc:492] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:25.315639 13874 ts_tablet_manager.cc:1397] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.002s
I20260807 08:00:25.315719 13876 ts_tablet_manager.cc:1397] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.006s
I20260807 08:00:25.317975 13875 tablet_bootstrap.cc:654] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.318704 13874 raft_consensus.cc:348] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.318984 13876 raft_consensus.cc:348] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.319630 13874 raft_consensus.cc:374] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.319905 13876 raft_consensus.cc:374] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.320012 13874 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.320411 13876 raft_consensus.cc:729] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.321267 13874 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.321303 13876 consensus_queue.cc:260] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.326249 13874 ts_tablet_manager.cc:1428] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20260807 08:00:25.326504 13876 ts_tablet_manager.cc:1428] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260807 08:00:25.327461 13874 tablet_bootstrap.cc:492] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:25.327996 13876 tablet_bootstrap.cc:492] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:25.334017 13876 tablet_bootstrap.cc:654] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.337900 13875 tablet_bootstrap.cc:492] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:25.338490 13875 ts_tablet_manager.cc:1397] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.004s
I20260807 08:00:25.343118 13875 raft_consensus.cc:348] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.343886 13875 raft_consensus.cc:374] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.344244 13875 raft_consensus.cc:729] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.344722 13876 tablet_bootstrap.cc:492] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:25.345085 13874 tablet_bootstrap.cc:654] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.345172 13876 ts_tablet_manager.cc:1397] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260807 08:00:25.346865 13432 catalog_manager.cc:5495] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:25.348024 13876 raft_consensus.cc:348] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } }
I20260807 08:00:25.348048 13875 consensus_queue.cc:260] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.348830 13876 raft_consensus.cc:374] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.349136 13876 raft_consensus.cc:729] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.349854 13876 consensus_queue.cc:260] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } }
I20260807 08:00:25.353724 13875 ts_tablet_manager.cc:1428] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260807 08:00:25.354658 13875 tablet_bootstrap.cc:492] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:25.359361 13876 ts_tablet_manager.cc:1428] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260807 08:00:25.360493 13875 tablet_bootstrap.cc:654] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.365362 13876 tablet_bootstrap.cc:492] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:25.368170 13874 tablet_bootstrap.cc:492] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:25.368621 13874 ts_tablet_manager.cc:1397] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.003s
I20260807 08:00:25.372116 13874 raft_consensus.cc:348] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.372928 13874 raft_consensus.cc:374] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.373227 13874 raft_consensus.cc:729] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.373951 13874 consensus_queue.cc:260] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.375029 13875 tablet_bootstrap.cc:492] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:25.375445 13875 ts_tablet_manager.cc:1397] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.008s
I20260807 08:00:25.376008 13874 ts_tablet_manager.cc:1428] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260807 08:00:25.377267 13875 raft_consensus.cc:348] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.377875 13875 raft_consensus.cc:374] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.378134 13875 raft_consensus.cc:729] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.378551 13874 tablet_bootstrap.cc:492] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:25.378770 13875 consensus_queue.cc:260] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.380705 13875 ts_tablet_manager.cc:1428] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260807 08:00:25.381834 13875 tablet_bootstrap.cc:492] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:25.381898 13876 tablet_bootstrap.cc:654] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.384974 13874 tablet_bootstrap.cc:654] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.387101 13875 tablet_bootstrap.cc:654] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.414000 13876 tablet_bootstrap.cc:492] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:25.414510 13876 ts_tablet_manager.cc:1397] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.049s user 0.024s sys 0.014s
I20260807 08:00:25.414837 13874 tablet_bootstrap.cc:492] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:25.415305 13874 ts_tablet_manager.cc:1397] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.016s
I20260807 08:00:25.415729 13875 tablet_bootstrap.cc:492] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:25.416175 13875 ts_tablet_manager.cc:1397] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.016s
I20260807 08:00:25.416941 13876 raft_consensus.cc:348] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.417552 13876 raft_consensus.cc:374] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.417853 13876 raft_consensus.cc:729] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.417989 13874 raft_consensus.cc:348] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.418773 13874 raft_consensus.cc:374] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.418560 13876 consensus_queue.cc:260] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.419090 13874 raft_consensus.cc:729] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.418617 13875 raft_consensus.cc:348] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.419508 13875 raft_consensus.cc:374] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.419770 13875 raft_consensus.cc:729] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.419790 13874 consensus_queue.cc:260] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.420624 13875 consensus_queue.cc:260] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.422183 13874 ts_tablet_manager.cc:1428] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 08:00:25.422566 13875 ts_tablet_manager.cc:1428] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260807 08:00:25.423344 13875 tablet_bootstrap.cc:492] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:25.423979 13876 ts_tablet_manager.cc:1428] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260807 08:00:25.424965 13876 tablet_bootstrap.cc:492] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:25.427237 13874 tablet_bootstrap.cc:492] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:25.428457 13875 tablet_bootstrap.cc:654] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.431785 13882 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.432217 13882 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.434139 13874 tablet_bootstrap.cc:654] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.434206 13882 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.435840 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:25.436599 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:00:25.437148 13657 leader_election.cc:341] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.437222 13591 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:25.438848 13661 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:25.441239 13876 tablet_bootstrap.cc:654] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.443992 13882 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:25.444343 13882 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:25.444671 13882 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.447269 13874 tablet_bootstrap.cc:492] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:25.447815 13874 ts_tablet_manager.cc:1397] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20260807 08:00:25.449729 13875 tablet_bootstrap.cc:492] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:25.450146 13875 ts_tablet_manager.cc:1397] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.012s
I20260807 08:00:25.450074 13874 raft_consensus.cc:348] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.450809 13874 raft_consensus.cc:374] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.451087 13874 raft_consensus.cc:729] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.451846 13874 consensus_queue.cc:260] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.454289 13912 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.454847 13912 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.455174 13875 raft_consensus.cc:348] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.455876 13875 raft_consensus.cc:374] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.456125 13875 raft_consensus.cc:729] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.456734 13874 ts_tablet_manager.cc:1428] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20260807 08:00:25.456707 13875 consensus_queue.cc:260] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.457679 13874 tablet_bootstrap.cc:492] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:25.459411 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:00:25.462239 13769 leader_election.cc:341] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.463896 13912 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:25.465368 13874 tablet_bootstrap.cc:654] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.465935 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:25.466773 13711 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:25.468364 13769 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:25.470165 13912 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:25.471222 13912 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:25.472172 13912 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.492476 13874 tablet_bootstrap.cc:492] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:25.493039 13874 ts_tablet_manager.cc:1397] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.017s
I20260807 08:00:25.496029 13874 raft_consensus.cc:348] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.496958 13874 raft_consensus.cc:374] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.497334 13874 raft_consensus.cc:729] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.498238 13874 consensus_queue.cc:260] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.501096 13874 ts_tablet_manager.cc:1428] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260807 08:00:25.502188 13874 tablet_bootstrap.cc:492] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:25.508786 13874 tablet_bootstrap.cc:654] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.515164 13959 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.518529 13922 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.519203 13922 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.521991 13922 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:25.524751 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:25.526656 13874 tablet_bootstrap.cc:492] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:25.526436 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:25.527397 13874 ts_tablet_manager.cc:1397] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.005s
W20260807 08:00:25.528565 13769 leader_election.cc:341] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:25.529460 13769 leader_election.cc:341] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.517859 13959 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.530011 13769 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:25.531396 13922 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:25.532902 13956 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.533308 13961 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.533421 13956 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.533763 13961 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } }
I20260807 08:00:25.534682 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:25.535481 13956 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.535517 13591 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:25.536087 13956 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.536518 13956 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.537021 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:25.537536 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:25.538074 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:25.538297 13882 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.538779 13816 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:25.539717 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:25.540213 13882 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.540865 13882 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:25.539886 13657 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:25.542068 13953 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:25.542575 13953 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:25.542953 13961 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:25.543073 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:25.543866 13816 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.542914 13953 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:25.550710 13661 leader_election.cc:341] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.552873 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:25.555094 13590 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:25.554037 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:25.556264 13592 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.556767 13884 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.557363 13884 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.560315 13884 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.562503 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:25.562937 13661 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:25.563432 13814 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:25.564286 13961 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:25.564639 13961 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:25.564695 13884 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.565033 13961 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.565624 13884 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.568198 13884 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.564767 13537 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:25.571235 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:25.573925 13711 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:25.572319 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:25.550874 13912 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.574457 13884 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:25.575184 13884 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:25.579468 13884 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.581697 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:25.541283 13882 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:25.575578 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:25.582540 13811 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:25.583418 13882 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:25.583693 13537 leader_election.cc:341] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.584451 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:25.585559 13912 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:00:25.585619 13769 leader_election.cc:341] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.576337 13956 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.574795 13710 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:25.572614 13961 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } }
I20260807 08:00:25.573031 13816 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:25.578665 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:25.589433 13590 raft_consensus.cc:2379] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 1: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:00:25.590317 13961 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:25.580015 13876 tablet_bootstrap.cc:492] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:25.591054 13876 ts_tablet_manager.cc:1397] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.166s user 0.043s sys 0.072s
I20260807 08:00:25.591892 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:25.592486 13590 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.593760 13876 raft_consensus.cc:348] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.594445 13876 raft_consensus.cc:374] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.594823 13876 raft_consensus.cc:729] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.594480 13882 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.580444 13959 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.594952 13884 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.531673 13874 raft_consensus.cc:348] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } }
I20260807 08:00:25.587572 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:25.587847 13875 ts_tablet_manager.cc:1428] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.137s user 0.035s sys 0.073s
I20260807 08:00:25.595914 13874 raft_consensus.cc:374] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.596092 13710 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.596230 13874 raft_consensus.cc:729] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.596938 13874 consensus_queue.cc:260] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } }
I20260807 08:00:25.588049 13540 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:25.577732 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:25.598173 13875 tablet_bootstrap.cc:492] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:25.597932 13876 consensus_queue.cc:260] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.581737 13953 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.605201 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:25.602157 13592 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:25.608970 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:25.615828 13874 ts_tablet_manager.cc:1428] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.088s user 0.006s sys 0.000s
I20260807 08:00:25.617894 13882 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.618659 13590 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:25.621335 13876 ts_tablet_manager.cc:1428] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20260807 08:00:25.622493 13876 tablet_bootstrap.cc:492] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:25.638211 13874 tablet_bootstrap.cc:492] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:25.641881 13816 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.616479 13884 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.643004 13884 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:25.643401 13884 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:25.643754 13884 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.644487 13875 tablet_bootstrap.cc:654] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.645955 13874 tablet_bootstrap.cc:654] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.645870 13953 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:00:25.646603 13661 leader_election.cc:341] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.647306 13661 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:25.649300 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:25.649935 13592 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.650367 13710 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:25.650369 13953 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.650990 13953 raft_consensus.cc:686] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:25.651803 13953 consensus_queue.cc:237] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } }
I20260807 08:00:25.652122 13657 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:25.653023 13882 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.649958 13816 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:25.653659 13817 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:25.654651 13769 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:25.656939 13922 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.657433 13883 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.642999 13876 tablet_bootstrap.cc:654] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.657410 13922 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.653281 13657 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:25.668931 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:25.669675 13710 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.670607 13882 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:25.673955 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:25.674728 13816 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.674844 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:25.674934 13592 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:25.675565 13812 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.678063 13537 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:25.683840 13882 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.684532 13882 raft_consensus.cc:686] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:25.686563 13922 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:25.688429 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:25.688627 13882 consensus_queue.cc:237] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.689603 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:25.676326 13883 raft_consensus.cc:686] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:25.691351 13951 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.692020 13951 raft_consensus.cc:686] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:25.685410 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:25.690054 13922 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:25.695065 13817 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:25.695979 13769 leader_election.cc:341] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.695092 13883 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.699194 13951 consensus_queue.cc:237] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.703009 13710 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:25.703578 13963 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.704102 13963 raft_consensus.cc:686] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:25.703106 13812 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:25.682101 13884 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.711077 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:25.711887 13711 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:25.712121 13657 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:25.708937 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:25.713421 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:25.715888 13710 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.705680 13963 consensus_queue.cc:237] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.716354 13922 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:25.716768 13922 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:25.717127 13922 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.720132 13953 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.720672 13953 raft_consensus.cc:686] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:25.713620 13884 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.725728 13428 catalog_manager.cc:5495] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:25.728147 13953 consensus_queue.cc:237] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.732162 13710 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:25.732283 13922 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.727617 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:00:25.734328 13537 leader_election.cc:341] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.734898 13875 tablet_bootstrap.cc:492] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:25.735325 13875 ts_tablet_manager.cc:1397] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.137s user 0.014s sys 0.033s
I20260807 08:00:25.737694 13540 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:25.747834 13817 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:25.751700 13876 tablet_bootstrap.cc:492] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:25.752172 13876 ts_tablet_manager.cc:1397] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.130s user 0.018s sys 0.017s
I20260807 08:00:25.752437 13883 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:25.754067 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:25.754624 13711 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.755326 13875 raft_consensus.cc:348] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.755971 13875 raft_consensus.cc:374] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.756209 13875 raft_consensus.cc:729] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.750556 13432 catalog_manager.cc:5495] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:25.746280 13963 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.757136 13963 raft_consensus.cc:686] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:25.757931 13963 consensus_queue.cc:237] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.750579 13922 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:25.758272 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:25.761266 13875 consensus_queue.cc:260] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.764050 13884 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.764230 13922 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:25.764696 13884 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.766870 13884 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.767257 13875 ts_tablet_manager.cc:1428] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.032s user 0.003s sys 0.006s
I20260807 08:00:25.768754 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:25.770100 13537 leader_election.cc:341] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.770283 13816 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:25.771829 13876 raft_consensus.cc:348] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.772498 13876 raft_consensus.cc:374] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.772535 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:25.772845 13876 raft_consensus.cc:729] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
W20260807 08:00:25.774487 13540 leader_election.cc:341] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.774945 13540 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:25.775736 13711 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
W20260807 08:00:25.776298 13769 leader_election.cc:341] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:25.768438 13875 tablet_bootstrap.cc:492] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:25.773447 13876 consensus_queue.cc:260] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.777668 13884 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:25.779659 13876 ts_tablet_manager.cc:1428] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20260807 08:00:25.780730 13876 tablet_bootstrap.cc:492] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:25.787992 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:25.765236 13431 catalog_manager.cc:5495] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:25.792166 13922 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:25.792851 13922 raft_consensus.cc:686] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:25.793679 13922 consensus_queue.cc:237] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.795074 13874 tablet_bootstrap.cc:492] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:25.795639 13874 ts_tablet_manager.cc:1397] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.158s user 0.015s sys 0.008s
I20260807 08:00:25.801750 13431 catalog_manager.cc:5495] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:25.802878 13432 catalog_manager.cc:5495] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:25.806847 13874 raft_consensus.cc:348] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.809703 13874 raft_consensus.cc:374] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:25.810102 13874 raft_consensus.cc:729] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:25.810658 13432 catalog_manager.cc:5495] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:25.827759 13876 tablet_bootstrap.cc:654] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:25.815022 13431 catalog_manager.cc:5495] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:25.842270 13428 catalog_manager.cc:5495] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:25.892484 13963 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:25.893206 13963 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.903216 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:25.904412 13709 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 1: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:00:25.906637 13963 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:25.907989 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:25.928934 13537 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:25.931167 13963 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:25.946295 13963 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:25.946206 13874 consensus_queue.cc:260] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:25.949651 13912 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:25.950548 13874 ts_tablet_manager.cc:1428] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.155s user 0.009s sys 0.001s
I20260807 08:00:25.951723 13874 tablet_bootstrap.cc:492] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:25.956080 13875 tablet_bootstrap.cc:654] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
W20260807 08:00:25.957170 13986 log.cc:924] Time spent T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260807 08:00:26.008214 13876 tablet_bootstrap.cc:492] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:26.008949 13876 ts_tablet_manager.cc:1397] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.228s user 0.026s sys 0.027s
I20260807 08:00:26.014346 13876 raft_consensus.cc:348] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.015156 13876 raft_consensus.cc:374] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.016281 13876 raft_consensus.cc:729] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.019275 13874 tablet_bootstrap.cc:654] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.019174 13876 consensus_queue.cc:260] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.021623 13876 ts_tablet_manager.cc:1428] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20260807 08:00:26.025995 13876 tablet_bootstrap.cc:492] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:26.026386 13874 tablet_bootstrap.cc:492] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:26.027237 13874 ts_tablet_manager.cc:1397] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.001s
I20260807 08:00:26.028568 13875 tablet_bootstrap.cc:492] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:26.028982 13875 ts_tablet_manager.cc:1397] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.261s user 0.008s sys 0.009s
I20260807 08:00:26.032469 13876 tablet_bootstrap.cc:654] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.031374 13875 raft_consensus.cc:348] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.034683 13875 raft_consensus.cc:374] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.034955 13875 raft_consensus.cc:729] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.035810 13875 consensus_queue.cc:260] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.038543 13875 ts_tablet_manager.cc:1428] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260807 08:00:26.039618 13875 tablet_bootstrap.cc:492] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:26.040686 13874 raft_consensus.cc:348] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.041383 13874 raft_consensus.cc:374] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.041671 13874 raft_consensus.cc:729] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.042390 13874 consensus_queue.cc:260] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.046883 13875 tablet_bootstrap.cc:654] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.048911 13951 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:26.052591 13874 ts_tablet_manager.cc:1428] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260807 08:00:26.059780 13875 tablet_bootstrap.cc:492] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:26.060393 13875 ts_tablet_manager.cc:1397] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.003s
I20260807 08:00:26.063580 13875 raft_consensus.cc:348] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.064465 13875 raft_consensus.cc:374] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.064785 13875 raft_consensus.cc:729] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.065052 13874 tablet_bootstrap.cc:492] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
W20260807 08:00:26.059749 13508 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 08:00:26.065593 13875 consensus_queue.cc:260] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.068540 13875 ts_tablet_manager.cc:1428] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260807 08:00:26.070096 13875 tablet_bootstrap.cc:492] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:26.077713 13875 tablet_bootstrap.cc:654] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.087157 13882 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:26.096029 13875 tablet_bootstrap.cc:492] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:26.096647 13875 ts_tablet_manager.cc:1397] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.008s
I20260807 08:00:26.099769 13875 raft_consensus.cc:348] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.100641 13875 raft_consensus.cc:374] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.100961 13875 raft_consensus.cc:729] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.101733 13875 consensus_queue.cc:260] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.105178 13875 ts_tablet_manager.cc:1428] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260807 08:00:26.106304 13875 tablet_bootstrap.cc:492] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:26.107070 13874 tablet_bootstrap.cc:654] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.113166 13875 tablet_bootstrap.cc:654] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.115113 13876 tablet_bootstrap.cc:492] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:26.115554 13876 ts_tablet_manager.cc:1397] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.090s user 0.026s sys 0.055s
I20260807 08:00:26.117983 13876 raft_consensus.cc:348] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.118698 13876 raft_consensus.cc:374] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.118938 13876 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.119656 13876 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.121811 13876 ts_tablet_manager.cc:1428] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260807 08:00:26.121789 13963 consensus_queue.cc:1035] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:26.122861 13876 tablet_bootstrap.cc:492] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:26.128867 13876 tablet_bootstrap.cc:654] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.137436 13963 consensus_queue.cc:1035] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:26.144276 13875 tablet_bootstrap.cc:492] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:26.144979 13875 ts_tablet_manager.cc:1397] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.004s
I20260807 08:00:26.152612 13874 tablet_bootstrap.cc:492] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:26.153213 13874 ts_tablet_manager.cc:1397] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.000s
I20260807 08:00:26.154731 13951 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.154893 13875 raft_consensus.cc:348] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.155988 13875 raft_consensus.cc:374] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.156333 13875 raft_consensus.cc:729] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.155241 13951 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.157123 13875 consensus_queue.cc:260] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.159155 13951 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:26.159395 13874 raft_consensus.cc:348] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.160039 13874 raft_consensus.cc:374] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.160149 13875 ts_tablet_manager.cc:1428] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20260807 08:00:26.160315 13874 raft_consensus.cc:729] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.160640 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:26.161337 13875 tablet_bootstrap.cc:492] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:26.162568 13874 consensus_queue.cc:260] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.167270 13874 ts_tablet_manager.cc:1428] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
W20260807 08:00:26.162849 13661 leader_election.cc:341] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:26.162691 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:26.163659 13963 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.169010 13963 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.168107 13874 tablet_bootstrap.cc:492] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:26.171365 13876 tablet_bootstrap.cc:492] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:26.171537 13963 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:26.172003 13876 ts_tablet_manager.cc:1397] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.049s user 0.019s sys 0.029s
W20260807 08:00:26.173208 13657 leader_election.cc:341] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:26.173688 13657 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:26.174912 13876 raft_consensus.cc:348] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.175403 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:26.176427 13883 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.176559 13874 tablet_bootstrap.cc:654] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.177618 13951 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:26.175689 13876 raft_consensus.cc:374] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.177307 13883 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.178220 13876 raft_consensus.cc:729] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.179332 13875 tablet_bootstrap.cc:654] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.179034 13876 consensus_queue.cc:260] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.179610 13883 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:26.181939 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:00:26.183625 13769 leader_election.cc:341] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:26.184983 13876 ts_tablet_manager.cc:1428] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260807 08:00:26.185923 13876 tablet_bootstrap.cc:492] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:26.186076 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:26.186910 13709 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:26.188640 13769 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:26.194557 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:26.195364 13817 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 1: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:00:26.196492 13537 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:26.197683 13876 tablet_bootstrap.cc:654] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.197799 13963 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.201941 13883 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:26.202452 13883 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:26.202881 13883 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.209930 13876 tablet_bootstrap.cc:492] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:26.210608 13876 ts_tablet_manager.cc:1397] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.013s
I20260807 08:00:26.211875 13884 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:26.214052 13951 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:26.213996 13876 raft_consensus.cc:348] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.215229 13876 raft_consensus.cc:374] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.215647 13876 raft_consensus.cc:729] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.216501 13876 consensus_queue.cc:260] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.219429 13876 ts_tablet_manager.cc:1428] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260807 08:00:26.220645 13876 tablet_bootstrap.cc:492] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:26.221958 13973 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.222528 13973 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.224905 13973 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:26.225596 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:26.226265 13711 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:26.226744 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:26.227347 13817 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:26.227504 13540 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:26.228320 13884 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:26.228662 13884 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:26.228996 13884 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.230278 13876 tablet_bootstrap.cc:654] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.231918 13912 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:26.233441 13971 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:26.237231 13884 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.239861 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:26.240469 13711 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.243610 13971 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.244239 13971 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.247643 13971 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:26.249083 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:26.249794 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:26.250440 13708 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
W20260807 08:00:26.250573 13769 leader_election.cc:341] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:26.247807 13711 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:26.251812 13769 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:26.252334 13540 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:26.253082 13973 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:26.253155 13971 raft_consensus.cc:2781] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:26.253868 13971 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:26.254386 13971 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.255904 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:26.256488 13816 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.257351 13884 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:26.250883 13963 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:26.264981 13971 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.267691 13953 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:26.286711 13883 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.292936 13971 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:26.277196 13951 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260807 08:00:26.290159 13973 raft_consensus.cc:686] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:26.306660 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:26.311757 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:00:26.312456 13769 leader_election.cc:341] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:26.313410 13769 leader_election.cc:341] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:26.314724 13589 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.315011 13922 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.315666 13922 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.308259 13882 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260807 08:00:26.318811 13922 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:26.320236 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:26.320865 13711 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.321409 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:26.299500 13883 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:26.328190 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:26.328771 13708 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.336243 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:26.342049 13875 tablet_bootstrap.cc:492] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:26.343510 13951 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.346314 13951 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.351672 13711 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:26.345011 13875 ts_tablet_manager.cc:1397] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.184s user 0.025s sys 0.045s
I20260807 08:00:26.365615 13883 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260807 08:00:26.362030 13882 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260807 08:00:26.344182 13973 consensus_queue.cc:237] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.377038 13882 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260807 08:00:26.385113 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:26.385807 13588 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:26.381003 13769 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:26.386941 13661 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:26.387056 13769 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:26.387770 13882 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:26.388237 13882 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:26.388554 13882 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.387753 13883 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:26.389665 13883 raft_consensus.cc:2781] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:26.390120 13883 raft_consensus.cc:686] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:26.390915 13883 consensus_queue.cc:237] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.391362 13708 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:26.392787 13817 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.407801 13769 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:26.409799 13971 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:26.405676 13951 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:26.410254 13971 raft_consensus.cc:686] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:26.418272 13876 tablet_bootstrap.cc:492] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:26.418723 13876 ts_tablet_manager.cc:1397] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.198s user 0.021s sys 0.082s
I20260807 08:00:26.415863 13971 consensus_queue.cc:237] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.413829 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:26.440613 13657 leader_election.cc:341] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:26.444396 13816 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:26.529876 13874 tablet_bootstrap.cc:492] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:26.654755 13884 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.655408 13884 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.659423 13875 raft_consensus.cc:348] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.660442 13875 raft_consensus.cc:374] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.660847 13875 raft_consensus.cc:729] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.661845 13875 consensus_queue.cc:260] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.659461 13874 ts_tablet_manager.cc:1397] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.492s user 0.044s sys 0.071s
I20260807 08:00:26.676324 13875 ts_tablet_manager.cc:1428] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.317s user 0.006s sys 0.001s
I20260807 08:00:26.678136 13875 tablet_bootstrap.cc:492] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:26.701685 13876 raft_consensus.cc:348] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.708190 13876 raft_consensus.cc:374] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.708616 13876 raft_consensus.cc:729] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.711151 13874 raft_consensus.cc:348] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.711705 13874 raft_consensus.cc:374] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.711941 13874 raft_consensus.cc:729] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.714622 13874 consensus_queue.cc:260] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.721661 13973 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:26.724483 13884 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:26.726013 13874 ts_tablet_manager.cc:1428] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.052s user 0.003s sys 0.003s
I20260807 08:00:26.726114 13876 consensus_queue.cc:260] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.732877 13874 tablet_bootstrap.cc:492] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
W20260807 08:00:26.726116 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 214774us
I20260807 08:00:26.738277 13876 ts_tablet_manager.cc:1428] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.319s user 0.007s sys 0.005s
I20260807 08:00:26.718976 13882 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.725481 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:26.767757 13876 tablet_bootstrap.cc:492] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:26.774420 13882 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:26.775183 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:26.775825 13592 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.786401 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:26.790104 13537 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:26.791851 13973 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:00:26.795429 13875 tablet_bootstrap.cc:654] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.803057 13874 tablet_bootstrap.cc:654] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.802996 13592 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:26.806236 13661 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:26.806731 14052 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:26.807236 13882 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:26.808823 13882 raft_consensus.cc:686] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:26.809696 13882 consensus_queue.cc:237] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.811669 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:26.812258 13811 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.812602 13876 tablet_bootstrap.cc:654] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.812450 13431 catalog_manager.cc:5495] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:26.856544 13811 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:26.857685 13875 tablet_bootstrap.cc:492] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:26.858239 13875 ts_tablet_manager.cc:1397] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.181s user 0.012s sys 0.002s
I20260807 08:00:26.861791 13875 raft_consensus.cc:348] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.862596 13875 raft_consensus.cc:374] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.863034 13875 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.863756 13875 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.867692 13874 tablet_bootstrap.cc:492] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:26.868271 13874 ts_tablet_manager.cc:1397] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.136s user 0.026s sys 0.015s
I20260807 08:00:26.870114 13875 ts_tablet_manager.cc:1428] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260807 08:00:26.871099 13875 tablet_bootstrap.cc:492] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:26.871214 13874 raft_consensus.cc:348] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.871855 13874 raft_consensus.cc:374] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.872107 13874 raft_consensus.cc:729] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.876200 13874 consensus_queue.cc:260] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.877851 13882 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.879112 14040 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.879477 13882 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.879573 14040 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.880331 13874 ts_tablet_manager.cc:1428] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260807 08:00:26.883270 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:26.881414 13882 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:26.883908 13874 tablet_bootstrap.cc:492] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:26.884568 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:26.881424 14040 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:26.886726 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:26.887068 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:26.887960 13657 leader_election.cc:341] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:26.888098 13661 leader_election.cc:341] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:26.888800 13661 leader_election.cc:341] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:26.888967 13657 leader_election.cc:341] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:26.889199 13661 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:26.889429 13657 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:26.890528 13882 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:26.890950 14040 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:26.893230 13973 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:26.906715 13876 tablet_bootstrap.cc:492] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:26.907164 13876 ts_tablet_manager.cc:1397] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.140s user 0.023s sys 0.029s
I20260807 08:00:26.908167 13875 tablet_bootstrap.cc:654] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.908572 13874 tablet_bootstrap.cc:654] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.913487 13876 raft_consensus.cc:348] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.914234 13876 raft_consensus.cc:374] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.914565 13876 raft_consensus.cc:729] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.915220 13876 consensus_queue.cc:260] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.917830 13876 ts_tablet_manager.cc:1428] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.010s user 0.001s sys 0.006s
I20260807 08:00:26.921810 13874 tablet_bootstrap.cc:492] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:26.918833 13876 tablet_bootstrap.cc:492] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:26.921942 13875 tablet_bootstrap.cc:492] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:26.922399 13874 ts_tablet_manager.cc:1397] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.008s
I20260807 08:00:26.922768 13875 ts_tablet_manager.cc:1397] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.003s
I20260807 08:00:26.927956 13874 raft_consensus.cc:348] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.928659 13874 raft_consensus.cc:374] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.928941 13874 raft_consensus.cc:729] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.929607 14028 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:26.946053 13874 consensus_queue.cc:260] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.947654 13592 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:26.948426 13874 ts_tablet_manager.cc:1428] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.025s user 0.013s sys 0.007s
I20260807 08:00:26.949363 13874 tablet_bootstrap.cc:492] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:26.951289 13876 tablet_bootstrap.cc:654] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.954875 13874 tablet_bootstrap.cc:654] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:26.958051 13971 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:26.962206 13430 catalog_manager.cc:5495] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:26.967509 13430 catalog_manager.cc:5495] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
W20260807 08:00:26.968338 13769 consensus_peers.cc:487] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Couldn't send request to peer 77da774f2bb54cf28302d53d0db224ad. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:26.971943 13971 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:26.973084 14052 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:26.974095 13875 raft_consensus.cc:348] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.973222 13971 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.974974 13875 raft_consensus.cc:374] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.982605 13875 raft_consensus.cc:729] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:26.983583 13875 consensus_queue.cc:260] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.986183 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:26.986753 13876 tablet_bootstrap.cc:492] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:26.987262 13876 ts_tablet_manager.cc:1397] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.010s
I20260807 08:00:26.990872 13876 raft_consensus.cc:348] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.991545 13876 raft_consensus.cc:374] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:26.991796 13876 raft_consensus.cc:729] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
W20260807 08:00:26.987927 13769 leader_election.cc:341] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:26.992358 13876 consensus_queue.cc:260] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:26.993629 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:00:26.995000 13769 leader_election.cc:341] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:26.995398 13769 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:26.996160 14052 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:26.999384 13875 ts_tablet_manager.cc:1428] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.076s user 0.008s sys 0.013s
I20260807 08:00:26.999671 13876 ts_tablet_manager.cc:1428] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20260807 08:00:27.000335 13875 tablet_bootstrap.cc:492] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.000723 13876 tablet_bootstrap.cc:492] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:26.998409 13971 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.007061 13874 tablet_bootstrap.cc:492] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.007633 13874 ts_tablet_manager.cc:1397] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.015s
I20260807 08:00:27.008035 13432 catalog_manager.cc:5495] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:27.013252 13874 raft_consensus.cc:348] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.014223 13874 raft_consensus.cc:374] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.014556 13874 raft_consensus.cc:729] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.015341 13874 consensus_queue.cc:260] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.020231 13874 ts_tablet_manager.cc:1428] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.012s user 0.008s sys 0.002s
I20260807 08:00:27.021953 13876 tablet_bootstrap.cc:654] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.026865 13874 tablet_bootstrap.cc:492] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.037300 14028 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.037878 14028 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.040197 14028 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.042065 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.044226 13875 tablet_bootstrap.cc:654] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.050952 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:00:27.050328 13537 leader_election.cc:341] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:27.055840 13540 leader_election.cc:341] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.053056 13971 consensus_queue.cc:1035] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260807 08:00:27.059868 13540 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:27.064019 14028 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:27.065469 13874 tablet_bootstrap.cc:654] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.089093 13876 tablet_bootstrap.cc:492] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.089732 13876 ts_tablet_manager.cc:1397] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.089s user 0.018s sys 0.012s
I20260807 08:00:27.094235 13882 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.094921 13882 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.097687 13882 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.099570 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.100345 13817 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:27.101540 13657 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.102540 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.103200 13592 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:27.104786 14040 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.105185 14040 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.105505 14040 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:27.106344 13508 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 08:00:27.110935 13876 raft_consensus.cc:348] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.111658 13876 raft_consensus.cc:374] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.111924 13876 raft_consensus.cc:729] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.114560 14040 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.123122 13875 tablet_bootstrap.cc:492] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.122866 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:27.123588 13875 ts_tablet_manager.cc:1397] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.123s user 0.010s sys 0.033s
I20260807 08:00:27.123620 13592 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.127820 13876 consensus_queue.cc:260] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.129494 13875 raft_consensus.cc:348] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.130246 13876 ts_tablet_manager.cc:1428] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260807 08:00:27.130262 13592 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:27.131461 13661 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.130318 13875 raft_consensus.cc:374] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.132257 13882 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.132526 13875 raft_consensus.cc:729] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.132809 13882 raft_consensus.cc:686] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:27.133782 13875 consensus_queue.cc:260] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.134397 14040 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.134799 13882 consensus_queue.cc:237] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.135609 13876 tablet_bootstrap.cc:492] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.136030 13875 ts_tablet_manager.cc:1428] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20260807 08:00:27.136840 13875 tablet_bootstrap.cc:492] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.133868 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:27.137681 13817 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.151870 13875 tablet_bootstrap.cc:654] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.149737 13817 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:27.157485 13876 tablet_bootstrap.cc:654] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.158927 13874 tablet_bootstrap.cc:492] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.159245 13874 ts_tablet_manager.cc:1397] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.133s user 0.017s sys 0.011s
I20260807 08:00:27.161419 14028 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.161885 14028 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.162644 13874 raft_consensus.cc:348] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.163552 13874 raft_consensus.cc:374] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.163806 14028 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.163862 13874 raft_consensus.cc:729] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.164788 13874 consensus_queue.cc:260] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.166293 13428 catalog_manager.cc:5495] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:27.168893 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.170823 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.172863 13874 ts_tablet_manager.cc:1428] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260807 08:00:27.174126 13874 tablet_bootstrap.cc:492] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.176715 13537 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:27.177922 14028 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:27.182296 13874 tablet_bootstrap.cc:654] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.187847 13875 tablet_bootstrap.cc:492] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.188277 13875 ts_tablet_manager.cc:1397] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.014s
I20260807 08:00:27.189018 13876 tablet_bootstrap.cc:492] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.189436 13876 ts_tablet_manager.cc:1397] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.005s
I20260807 08:00:27.190802 13875 raft_consensus.cc:348] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.191650 13875 raft_consensus.cc:374] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.191977 13875 raft_consensus.cc:729] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.192768 13875 consensus_queue.cc:260] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.192212 13876 raft_consensus.cc:348] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.193982 13876 raft_consensus.cc:374] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.196341 13876 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.196899 13876 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.198642 13874 tablet_bootstrap.cc:492] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.198860 13875 ts_tablet_manager.cc:1428] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260807 08:00:27.199040 13876 ts_tablet_manager.cc:1428] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260807 08:00:27.199055 13874 ts_tablet_manager.cc:1397] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.006s
I20260807 08:00:27.199667 13875 tablet_bootstrap.cc:492] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.199990 13876 tablet_bootstrap.cc:492] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.201498 13874 raft_consensus.cc:348] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.202136 13874 raft_consensus.cc:374] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.202392 13874 raft_consensus.cc:729] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.205027 13882 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.205461 13882 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.203075 13874 consensus_queue.cc:260] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.207338 13882 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.209002 13874 ts_tablet_manager.cc:1428] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.010s user 0.001s sys 0.003s
I20260807 08:00:27.209084 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.209450 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:27.210435 13661 leader_election.cc:341] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.214022 13875 tablet_bootstrap.cc:654] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.216172 13874 tablet_bootstrap.cc:492] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
W20260807 08:00:27.217875 13657 leader_election.cc:341] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.218995 13876 tablet_bootstrap.cc:654] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.219738 13657 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:27.220991 13882 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:27.224690 13874 tablet_bootstrap.cc:654] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.240306 13874 tablet_bootstrap.cc:492] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.240736 13874 ts_tablet_manager.cc:1397] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.009s
I20260807 08:00:27.242306 13876 tablet_bootstrap.cc:492] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.243736 13875 tablet_bootstrap.cc:492] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.243584 13874 raft_consensus.cc:348] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.245841 13875 ts_tablet_manager.cc:1397] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.005s
I20260807 08:00:27.245869 13876 ts_tablet_manager.cc:1397] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.008s
I20260807 08:00:27.248952 13875 raft_consensus.cc:348] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.249886 13875 raft_consensus.cc:374] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.250206 13875 raft_consensus.cc:729] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.249631 13876 raft_consensus.cc:348] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.250558 13876 raft_consensus.cc:374] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.250845 13876 raft_consensus.cc:729] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.250815 13875 consensus_queue.cc:260] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.251626 13876 consensus_queue.cc:260] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.252805 13875 ts_tablet_manager.cc:1428] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260807 08:00:27.253124 13874 raft_consensus.cc:374] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.253377 13874 raft_consensus.cc:729] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.254073 13874 consensus_queue.cc:260] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.254982 13876 ts_tablet_manager.cc:1428] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260807 08:00:27.256424 13876 tablet_bootstrap.cc:492] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.261724 13876 tablet_bootstrap.cc:654] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.264540 13874 ts_tablet_manager.cc:1428] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260807 08:00:27.265654 13875 tablet_bootstrap.cc:492] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.281651 13874 tablet_bootstrap.cc:492] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.292045 14040 consensus_queue.cc:1035] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:27.302021 13875 tablet_bootstrap.cc:654] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.302349 13874 tablet_bootstrap.cc:654] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.302757 13876 tablet_bootstrap.cc:492] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.303272 13876 ts_tablet_manager.cc:1397] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.013s
I20260807 08:00:27.305907 13876 raft_consensus.cc:348] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.306779 13876 raft_consensus.cc:374] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.307107 13876 raft_consensus.cc:729] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.309993 14040 consensus_queue.cc:1035] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:27.322120 14040 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.322662 14040 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.321306 13876 consensus_queue.cc:260] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.325965 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.326532 13592 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:27.326772 13876 ts_tablet_manager.cc:1428] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260807 08:00:27.328158 13876 tablet_bootstrap.cc:492] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.328111 13661 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.329041 13951 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.329388 13951 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.329732 13951 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.347517 13951 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.353892 13876 tablet_bootstrap.cc:654] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.354519 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:27.355161 13590 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.358042 13884 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.358530 13884 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.360560 13884 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.362287 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.362979 13709 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:27.364221 13540 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.366194 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:27.367722 13537 leader_election.cc:341] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.372851 13875 tablet_bootstrap.cc:492] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.373278 13875 ts_tablet_manager.cc:1397] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.108s user 0.033s sys 0.025s
I20260807 08:00:27.373333 13963 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.373692 13876 tablet_bootstrap.cc:492] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.373765 13963 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.374209 13876 ts_tablet_manager.cc:1397] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.046s user 0.022s sys 0.000s
I20260807 08:00:27.374619 13884 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.349866 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.375408 13814 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:27.375808 13875 raft_consensus.cc:348] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.376514 13875 raft_consensus.cc:374] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.376770 13875 raft_consensus.cc:729] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.374228 13963 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.376670 13876 raft_consensus.cc:348] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.377475 13876 raft_consensus.cc:374] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.377830 13876 raft_consensus.cc:729] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.377578 13875 consensus_queue.cc:260] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.379832 13875 ts_tablet_manager.cc:1428] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 08:00:27.376979 13884 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.378474 13876 consensus_queue.cc:260] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.382300 13884 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.349639 14040 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.382820 13876 ts_tablet_manager.cc:1428] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260807 08:00:27.383735 13876 tablet_bootstrap.cc:492] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.383949 13590 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:27.384059 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.384696 13814 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:27.385151 13661 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.385982 13537 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:27.386109 14040 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.386780 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.387425 13709 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:27.389411 13874 tablet_bootstrap.cc:492] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.394671 13874 ts_tablet_manager.cc:1397] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.114s user 0.005s sys 0.004s
I20260807 08:00:27.397576 13874 raft_consensus.cc:348] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.387038 13875 tablet_bootstrap.cc:492] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.403872 13876 tablet_bootstrap.cc:654] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.393821 13951 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.398744 14040 raft_consensus.cc:686] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:27.409250 14040 consensus_queue.cc:237] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.411286 13874 raft_consensus.cc:374] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.399981 13883 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.413014 13883 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.415138 13883 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.398978 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:27.416517 13814 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.416954 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.417506 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.419421 13590 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:27.420436 13769 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:27.417840 13709 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:27.391601 13884 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.422245 13884 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.421322 13883 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.422730 13884 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.422845 13883 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.423328 13883 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.429463 13971 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.429988 13971 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.390864 13963 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.431902 13971 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.432379 13883 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.433475 13963 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.434149 13874 raft_consensus.cc:729] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.435293 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:27.435727 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.435886 13710 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.436350 13709 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:27.437402 13769 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.438218 13971 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.438548 13971 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.438858 13971 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.442189 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:00:27.443342 13769 leader_election.cc:341] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.443929 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:27.444916 13710 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
W20260807 08:00:27.445125 13537 leader_election.cc:341] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.446247 13540 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:27.448722 13874 consensus_queue.cc:260] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.450409 14111 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.451247 13874 ts_tablet_manager.cc:1428] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.056s user 0.005s sys 0.000s
I20260807 08:00:27.452096 13874 tablet_bootstrap.cc:492] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.452695 13883 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.412609 13875 tablet_bootstrap.cc:654] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.453679 13971 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.455598 13971 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.431918 13814 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:27.456359 14111 raft_consensus.cc:686] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:27.457259 14111 consensus_queue.cc:237] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.459062 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:27.459640 13708 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.461858 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:27.462556 13876 tablet_bootstrap.cc:492] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.462512 13590 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.463048 13876 ts_tablet_manager.cc:1397] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.079s user 0.016s sys 0.000s
I20260807 08:00:27.463217 13874 tablet_bootstrap.cc:654] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.465405 13876 raft_consensus.cc:348] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.465618 13884 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.466171 13876 raft_consensus.cc:374] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.466514 13876 raft_consensus.cc:729] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.467113 13876 consensus_queue.cc:260] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.469406 13875 tablet_bootstrap.cc:492] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.467444 13708 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:27.474161 13875 ts_tablet_manager.cc:1397] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.087s user 0.008s sys 0.004s
I20260807 08:00:27.474295 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:27.474807 13883 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.474896 13592 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.475108 13876 ts_tablet_manager.cc:1428] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.012s user 0.009s sys 0.001s
I20260807 08:00:27.475330 13883 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.475970 13876 tablet_bootstrap.cc:492] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.476573 13875 raft_consensus.cc:348] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.477160 13883 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.467628 13884 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.471480 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:27.479338 13710 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.482758 13592 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:27.471066 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:27.486316 13814 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.486905 13769 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.488006 13971 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.488282 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.470587 13431 catalog_manager.cc:5495] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:27.491150 13708 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:27.495260 13589 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.495199 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.498785 13769 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.500767 13883 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.501114 13883 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.501452 13883 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.503326 13590 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:27.488494 13971 raft_consensus.cc:686] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:27.507634 13971 consensus_queue.cc:237] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.477233 13875 raft_consensus.cc:374] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.490150 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:27.512532 13709 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.512786 13883 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.515028 13883 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.512796 13814 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:27.514868 13710 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:27.516453 13537 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:27.517813 13963 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.518400 13963 raft_consensus.cc:686] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:27.518870 13709 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:27.488612 13876 tablet_bootstrap.cc:654] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.519860 13963 consensus_queue.cc:237] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.520339 13875 raft_consensus.cc:729] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.523936 13875 consensus_queue.cc:260] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.528124 13874 tablet_bootstrap.cc:492] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.528625 13874 ts_tablet_manager.cc:1397] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.013s
I20260807 08:00:27.529009 13875 ts_tablet_manager.cc:1428] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.055s user 0.004s sys 0.003s
I20260807 08:00:27.524659 13431 catalog_manager.cc:5495] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:27.529927 13875 tablet_bootstrap.cc:492] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.531559 13874 raft_consensus.cc:348] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.534246 13874 raft_consensus.cc:374] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.534919 13874 raft_consensus.cc:729] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.536168 13875 tablet_bootstrap.cc:654] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.535938 13874 consensus_queue.cc:260] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.538482 13874 ts_tablet_manager.cc:1428] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260807 08:00:27.539345 13874 tablet_bootstrap.cc:492] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.544199 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:27.544795 13709 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.546075 13874 tablet_bootstrap.cc:654] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.547138 13876 tablet_bootstrap.cc:492] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.547119 13769 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.547714 13876 ts_tablet_manager.cc:1397] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.072s user 0.007s sys 0.003s
I20260807 08:00:27.549691 13971 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.550174 13971 raft_consensus.cc:686] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:27.550761 13876 raft_consensus.cc:348] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.550897 13971 consensus_queue.cc:237] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.552644 13709 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:27.553722 13876 raft_consensus.cc:374] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.554111 13876 raft_consensus.cc:729] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.553192 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:00:27.553844 13769 leader_election.cc:341] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.554850 13876 consensus_queue.cc:260] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:27.557552 13769 leader_election.cc:341] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.558337 13876 ts_tablet_manager.cc:1428] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260807 08:00:27.558472 13769 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:27.559329 13876 tablet_bootstrap.cc:492] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.556488 13428 catalog_manager.cc:5495] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:27.559500 13883 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.560182 13883 raft_consensus.cc:686] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:27.561260 13883 consensus_queue.cc:237] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.571363 13874 tablet_bootstrap.cc:492] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.571497 13876 tablet_bootstrap.cc:654] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.571815 13874 ts_tablet_manager.cc:1397] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.001s
I20260807 08:00:27.574437 13874 raft_consensus.cc:348] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.575095 13874 raft_consensus.cc:374] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.575353 13874 raft_consensus.cc:729] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.575904 13874 consensus_queue.cc:260] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.577838 13874 ts_tablet_manager.cc:1428] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260807 08:00:27.586480 13874 tablet_bootstrap.cc:492] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.590858 13875 tablet_bootstrap.cc:492] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.590171 13430 catalog_manager.cc:5495] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:27.591643 13875 ts_tablet_manager.cc:1397] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.012s
I20260807 08:00:27.591823 13874 tablet_bootstrap.cc:654] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.592484 13430 catalog_manager.cc:5495] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:27.596910 13875 raft_consensus.cc:348] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.597687 13875 raft_consensus.cc:374] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.597949 13875 raft_consensus.cc:729] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.598551 13875 consensus_queue.cc:260] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.601613 13875 ts_tablet_manager.cc:1428] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260807 08:00:27.602595 13875 tablet_bootstrap.cc:492] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.606812 13876 tablet_bootstrap.cc:492] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.607371 13876 ts_tablet_manager.cc:1397] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.048s user 0.017s sys 0.017s
I20260807 08:00:27.608026 13875 tablet_bootstrap.cc:654] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.613938 13875 tablet_bootstrap.cc:492] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.614353 13875 ts_tablet_manager.cc:1397] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260807 08:00:27.615079 13876 raft_consensus.cc:348] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.615830 13876 raft_consensus.cc:374] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.616073 13876 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.616570 13875 raft_consensus.cc:348] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.616752 13876 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.616360 13430 catalog_manager.cc:5495] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:27.617297 13875 raft_consensus.cc:374] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.617626 13875 raft_consensus.cc:729] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.618184 13875 consensus_queue.cc:260] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.618863 13876 ts_tablet_manager.cc:1428] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260807 08:00:27.620067 13875 ts_tablet_manager.cc:1428] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260807 08:00:27.620270 13874 tablet_bootstrap.cc:492] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.620642 13874 ts_tablet_manager.cc:1397] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.008s
I20260807 08:00:27.620918 13875 tablet_bootstrap.cc:492] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.623476 13874 raft_consensus.cc:348] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.624125 13874 raft_consensus.cc:374] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.624380 13874 raft_consensus.cc:729] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.625012 13874 consensus_queue.cc:260] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.627084 13874 ts_tablet_manager.cc:1428] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260807 08:00:27.627192 13875 tablet_bootstrap.cc:654] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.627754 13971 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.628000 13874 tablet_bootstrap.cc:492] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.628191 13971 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.630033 13876 tablet_bootstrap.cc:492] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.631702 13951 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:27.638219 13876 tablet_bootstrap.cc:654] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.639827 13963 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.640424 13963 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.643052 13874 tablet_bootstrap.cc:654] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.643728 13971 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.646081 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:00:27.649609 13769 leader_election.cc:341] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.649816 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.652750 14113 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.653244 14113 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.654688 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.655228 14113 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.655514 13589 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:27.656612 13769 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:00:27.657723 13537 leader_election.cc:341] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.658026 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.658648 13709 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:27.658949 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.659545 13874 tablet_bootstrap.cc:492] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.659741 13540 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.660668 13971 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.660949 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.659981 13874 ts_tablet_manager.cc:1397] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.004s
I20260807 08:00:27.661948 13963 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.662067 14113 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.664383 14113 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.664852 14113 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.661209 13971 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.659746 13814 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:27.670028 13971 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:27.663570 13540 leader_election.cc:341] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.671520 13540 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:27.672255 14111 raft_consensus.cc:2726] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:27.675091 13875 tablet_bootstrap.cc:492] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.675557 13875 ts_tablet_manager.cc:1397] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.037s
I20260807 08:00:27.676913 13971 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.677711 14113 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.677983 13875 raft_consensus.cc:348] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.678675 13875 raft_consensus.cc:374] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.678932 13875 raft_consensus.cc:729] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.679549 13875 consensus_queue.cc:260] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.679987 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:27.680586 13589 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.680804 14113 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.681658 13874 raft_consensus.cc:348] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.682406 13874 raft_consensus.cc:374] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.682664 13874 raft_consensus.cc:729] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.683432 13874 consensus_queue.cc:260] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.685689 13874 ts_tablet_manager.cc:1428] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.024s user 0.004s sys 0.002s
I20260807 08:00:27.680586 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:27.686592 13814 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.687978 13874 tablet_bootstrap.cc:492] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.691824 13875 ts_tablet_manager.cc:1428] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s
I20260807 08:00:27.691740 14105 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:27.692757 13875 tablet_bootstrap.cc:492] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.698210 13875 tablet_bootstrap.cc:654] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.698269 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:27.698911 13710 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.703564 13874 tablet_bootstrap.cc:654] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.709023 13589 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:27.712669 13769 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:27.713471 13883 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.713941 13883 raft_consensus.cc:686] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:27.714732 13883 consensus_queue.cc:237] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.723520 14113 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.711264 13814 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:27.724051 14113 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.724309 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:27.725035 13971 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.726547 14113 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.725746 13709 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.729251 13537 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:27.730727 13963 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.731287 13963 raft_consensus.cc:686] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:27.732095 13963 consensus_queue.cc:237] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.739471 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.740015 13709 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:27.740633 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:27.741945 13537 leader_election.cc:341] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.711206 13710 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
W20260807 08:00:27.755632 13540 leader_election.cc:341] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.756121 13540 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:27.756996 14113 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:27.758946 13430 catalog_manager.cc:5495] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:27.760352 13876 tablet_bootstrap.cc:492] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.760872 13876 ts_tablet_manager.cc:1397] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.131s user 0.013s sys 0.014s
I20260807 08:00:27.762622 13875 tablet_bootstrap.cc:492] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.763054 13875 ts_tablet_manager.cc:1397] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.070s user 0.016s sys 0.012s
I20260807 08:00:27.763327 13874 tablet_bootstrap.cc:492] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.763825 13874 ts_tablet_manager.cc:1397] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.076s user 0.029s sys 0.016s
I20260807 08:00:27.765292 13875 raft_consensus.cc:348] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.765993 13875 raft_consensus.cc:374] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.766265 13875 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.766902 13875 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.769022 13875 ts_tablet_manager.cc:1428] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 08:00:27.764087 13876 raft_consensus.cc:348] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.769369 13874 raft_consensus.cc:348] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.770246 13874 raft_consensus.cc:374] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.770505 13874 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.769897 13876 raft_consensus.cc:374] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.771113 13876 raft_consensus.cc:729] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.771157 13874 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.771823 13876 consensus_queue.cc:260] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.773082 13874 ts_tablet_manager.cc:1428] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260807 08:00:27.774266 13875 tablet_bootstrap.cc:492] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.773828 13876 ts_tablet_manager.cc:1428] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.013s user 0.005s sys 0.003s
I20260807 08:00:27.775405 13876 tablet_bootstrap.cc:492] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.777776 13874 tablet_bootstrap.cc:492] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.780090 13875 tablet_bootstrap.cc:654] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.780422 13876 tablet_bootstrap.cc:654] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.785554 13874 tablet_bootstrap.cc:654] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.792162 13875 tablet_bootstrap.cc:492] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.792644 13875 ts_tablet_manager.cc:1397] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.012s
I20260807 08:00:27.795132 13875 raft_consensus.cc:348] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.795845 13875 raft_consensus.cc:374] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.796191 13875 raft_consensus.cc:729] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.788344 13432 catalog_manager.cc:5495] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:27.796960 13875 consensus_queue.cc:260] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.799551 13875 ts_tablet_manager.cc:1428] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 08:00:27.800467 13875 tablet_bootstrap.cc:492] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.806085 13875 tablet_bootstrap.cc:654] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.812889 13875 tablet_bootstrap.cc:492] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.813359 13875 ts_tablet_manager.cc:1397] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260807 08:00:27.817328 13875 raft_consensus.cc:348] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.818140 14105 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.818590 13875 raft_consensus.cc:374] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.819070 13875 raft_consensus.cc:729] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.818711 14105 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.819934 13875 consensus_queue.cc:260] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.822223 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.822897 13814 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:27.823535 13875 ts_tablet_manager.cc:1428] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260807 08:00:27.824150 13657 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.825042 13951 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.825172 13875 tablet_bootstrap.cc:492] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.825462 13951 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.827149 13876 tablet_bootstrap.cc:492] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.830092 13951 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.830426 13876 ts_tablet_manager.cc:1397] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.001s
I20260807 08:00:27.831637 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.832324 13589 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:27.833441 14105 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.835391 13874 tablet_bootstrap.cc:492] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.835916 13874 ts_tablet_manager.cc:1397] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.058s user 0.027s sys 0.009s
I20260807 08:00:27.838529 13874 raft_consensus.cc:348] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.839712 13874 raft_consensus.cc:374] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.840081 13874 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.833375 13876 raft_consensus.cc:348] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.840891 13876 raft_consensus.cc:374] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.841251 13876 raft_consensus.cc:729] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.842075 13876 consensus_queue.cc:260] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.843204 13875 tablet_bootstrap.cc:654] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.844024 13874 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.846455 13951 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.845768 13876 ts_tablet_manager.cc:1428] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260807 08:00:27.847975 13876 tablet_bootstrap.cc:492] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.848804 13951 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.849785 13874 ts_tablet_manager.cc:1428] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260807 08:00:27.855873 13874 tablet_bootstrap.cc:492] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.857582 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:27.857410 14111 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:27.858322 13589 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.861318 13963 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.854910 13876 tablet_bootstrap.cc:654] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.861917 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:27.862617 13814 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.862057 13963 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.865214 13874 tablet_bootstrap.cc:654] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.868413 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.869653 13705 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 1: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:00:27.872678 13963 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.875314 14105 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:27.878948 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.885169 13537 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:27.887107 14113 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.899168 13814 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:27.902323 13589 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:27.902715 14104 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.903278 14104 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.903824 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.905161 14105 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:27.905850 14105 raft_consensus.cc:686] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:27.906899 14113 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:27.914674 13874 tablet_bootstrap.cc:492] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.915428 13874 ts_tablet_manager.cc:1397] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.060s user 0.017s sys 0.014s
I20260807 08:00:27.906982 14105 consensus_queue.cc:237] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.923589 13874 raft_consensus.cc:348] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.925284 13874 raft_consensus.cc:374] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.926043 13874 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.927181 13874 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.917936 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.929822 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.921835 14104 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:00:27.931610 13661 leader_election.cc:341] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.933691 13874 ts_tablet_manager.cc:1428] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260807 08:00:27.935411 13874 tablet_bootstrap.cc:492] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.935701 14104 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.936331 14104 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:27.935277 13657 leader_election.cc:341] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.938313 13657 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:27.938541 14104 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.940632 14040 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:27.942495 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:27.943347 13874 tablet_bootstrap.cc:654] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.943246 13814 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:27.945680 14104 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.946158 13876 tablet_bootstrap.cc:492] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:27.946659 13876 ts_tablet_manager.cc:1397] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.099s user 0.017s sys 0.021s
I20260807 08:00:27.946885 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.947786 13875 tablet_bootstrap.cc:492] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:27.947671 13883 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:27.948276 13875 ts_tablet_manager.cc:1397] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.123s user 0.050s sys 0.035s
I20260807 08:00:27.948292 13883 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.949971 13814 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.950615 13883 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:27.949302 13876 raft_consensus.cc:348] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.952121 13876 raft_consensus.cc:374] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.952414 13876 raft_consensus.cc:729] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.953089 13876 consensus_queue.cc:260] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.953035 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:27.954784 13590 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
W20260807 08:00:27.955302 13769 leader_election.cc:341] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.948302 13657 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:27.957353 14040 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:27.957783 14040 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:27.958149 14040 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.959115 13876 ts_tablet_manager.cc:1428] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260807 08:00:27.954785 13875 raft_consensus.cc:348] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.960053 13876 tablet_bootstrap.cc:492] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:27.960022 13875 raft_consensus.cc:374] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.960634 13875 raft_consensus.cc:729] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.961437 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:27.946188 14104 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:27.963222 13769 leader_election.cc:341] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.963687 13769 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:27.964845 13883 raft_consensus.cc:2726] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:27.966379 13876 tablet_bootstrap.cc:654] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:27.966200 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:00:27.967610 13661 leader_election.cc:341] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:27.961269 13875 consensus_queue.cc:260] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.971329 13875 ts_tablet_manager.cc:1428] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260807 08:00:27.973114 14113 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:27.974378 13875 tablet_bootstrap.cc:492] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:27.972698 14040 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.981393 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:27.982142 13814 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.982575 14040 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:27.983353 13874 tablet_bootstrap.cc:492] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:27.983844 13874 ts_tablet_manager.cc:1397] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.049s user 0.013s sys 0.003s
I20260807 08:00:27.986505 13874 raft_consensus.cc:348] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.987226 13874 raft_consensus.cc:374] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:27.987545 13874 raft_consensus.cc:729] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:27.988245 13874 consensus_queue.cc:260] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:27.990685 13874 ts_tablet_manager.cc:1428] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 08:00:27.991613 13874 tablet_bootstrap.cc:492] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:27.994119 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:27.994895 13589 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:27.997565 13874 tablet_bootstrap.cc:654] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.000845 13876 tablet_bootstrap.cc:492] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:28.001310 13876 ts_tablet_manager.cc:1397] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.009s
I20260807 08:00:28.002138 13883 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.004186 13876 raft_consensus.cc:348] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.004945 13876 raft_consensus.cc:374] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.005232 13876 raft_consensus.cc:729] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.005939 13876 consensus_queue.cc:260] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.008893 13951 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.012082 13883 consensus_queue.cc:1035] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.020988 13883 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.023999 13875 tablet_bootstrap.cc:654] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.023728 14113 consensus_queue.cc:1035] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.025231 13589 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.034592 13876 ts_tablet_manager.cc:1428] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260807 08:00:28.036891 13876 tablet_bootstrap.cc:492] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:28.039678 13875 tablet_bootstrap.cc:492] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:28.040270 13875 ts_tablet_manager.cc:1397] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.002s
I20260807 08:00:28.041241 13661 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.042358 14040 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.042871 14040 raft_consensus.cc:686] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:28.035130 13814 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.046734 13883 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.055763 14104 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.061316 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:28.067104 13657 leader_election.cc:341] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.067737 13657 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:28.069104 14049 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.069746 13971 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.069655 14049 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.071957 14049 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:28.076860 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.077625 13591 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:28.083139 13769 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:28.084167 13971 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.084307 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:28.084709 13971 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.085237 13971 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.062858 13876 tablet_bootstrap.cc:654] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.085137 13711 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:28.059476 14040 consensus_queue.cc:237] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.089390 13874 tablet_bootstrap.cc:492] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:28.092597 13874 ts_tablet_manager.cc:1397] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.101s user 0.018s sys 0.048s
I20260807 08:00:28.094015 14105 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:28.094759 14105 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.095297 14105 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.097005 13876 tablet_bootstrap.cc:492] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:28.097507 13876 ts_tablet_manager.cc:1397] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.005s
I20260807 08:00:28.097438 14105 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.099530 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.100272 13591 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.100252 13876 raft_consensus.cc:348] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.101789 14105 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.097011 13875 raft_consensus.cc:348] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.102741 13875 raft_consensus.cc:374] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.103178 13875 raft_consensus.cc:729] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.102696 14105 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.103873 13875 consensus_queue.cc:260] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.105397 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:28.101145 13876 raft_consensus.cc:374] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.105425 14105 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.106758 13876 raft_consensus.cc:729] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.107194 13814 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.106662 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:28.107476 13876 consensus_queue.cc:260] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.108925 13657 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.106694 13874 raft_consensus.cc:348] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.110385 14105 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.111274 13874 raft_consensus.cc:374] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.110989 14105 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.106282 13875 ts_tablet_manager.cc:1428] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.066s user 0.005s sys 0.002s
I20260807 08:00:28.111753 13874 raft_consensus.cc:729] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.112051 14105 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.114301 14113 consensus_queue.cc:1035] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.114785 13876 ts_tablet_manager.cc:1428] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260807 08:00:28.116587 13875 tablet_bootstrap.cc:492] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:28.112496 13874 consensus_queue.cc:260] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.117971 13816 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.115761 13876 tablet_bootstrap.cc:492] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:28.125656 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.126386 13589 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.128818 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.135030 14105 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.135340 13971 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.136411 13874 ts_tablet_manager.cc:1428] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.043s user 0.012s sys 0.005s
I20260807 08:00:28.137475 13874 tablet_bootstrap.cc:492] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:28.138051 14040 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.138476 14040 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.139194 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:28.138849 14040 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.139853 13814 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.140425 13876 tablet_bootstrap.cc:654] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.143505 13874 tablet_bootstrap.cc:654] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.143653 13875 tablet_bootstrap.cc:654] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
W20260807 08:00:28.145759 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 62741.3us
I20260807 08:00:28.153748 13814 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.156247 13657 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.157550 14104 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.158236 14104 raft_consensus.cc:686] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:28.159116 13875 tablet_bootstrap.cc:492] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:28.159716 13875 ts_tablet_manager.cc:1397] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.005s
I20260807 08:00:28.159296 14104 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.166955 13876 tablet_bootstrap.cc:492] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:28.167454 13876 ts_tablet_manager.cc:1397] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.000s
I20260807 08:00:28.175794 13432 catalog_manager.cc:5495] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:28.154557 14105 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:00:28.185880 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 57187us
I20260807 08:00:28.187683 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:28.188390 13592 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.191354 13875 raft_consensus.cc:348] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.194176 13875 raft_consensus.cc:374] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.194506 13875 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.195390 13874 tablet_bootstrap.cc:492] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:28.195251 13875 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.196069 13874 ts_tablet_manager.cc:1397] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.059s user 0.022s sys 0.031s
W20260807 08:00:28.198026 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 50568.5us
I20260807 08:00:28.200133 13875 ts_tablet_manager.cc:1428] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.040s user 0.007s sys 0.002s
I20260807 08:00:28.201114 13875 tablet_bootstrap.cc:492] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:28.201323 13592 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.202919 13876 raft_consensus.cc:348] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.203922 13876 raft_consensus.cc:374] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.204268 13876 raft_consensus.cc:729] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.205175 13876 consensus_queue.cc:260] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.209631 13971 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:28.223587 13951 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.224103 13951 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.226327 13951 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.229074 14185 consensus_queue.cc:1035] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.231817 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:28.231940 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.232528 13709 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.227516 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:28.234440 13816 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.237874 13591 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.238221 13657 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.239194 13951 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.239758 13951 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.240269 13951 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.243301 14111 consensus_queue.cc:1035] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.243243 14040 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.245455 14040 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.246809 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:28.247462 13816 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.247952 13874 raft_consensus.cc:348] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.248723 13874 raft_consensus.cc:374] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.249008 13874 raft_consensus.cc:729] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.249677 13874 consensus_queue.cc:260] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.251157 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:28.251827 13589 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.251962 13874 ts_tablet_manager.cc:1428] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.055s user 0.004s sys 0.000s
I20260807 08:00:28.252926 13874 tablet_bootstrap.cc:492] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:28.257227 13709 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:28.260627 13875 tablet_bootstrap.cc:654] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.259881 13431 catalog_manager.cc:5495] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:28.254710 13816 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.271859 13431 catalog_manager.cc:5495] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:28.254756 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:28.287515 13589 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.289088 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.290438 14040 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.317488 14049 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.327696 14049 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.329929 14049 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:28.315711 14040 raft_consensus.cc:686] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:28.364686 14040 consensus_queue.cc:237] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:28.368642 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 47251.6us
I20260807 08:00:28.389894 13951 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.393870 13769 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.394735 14049 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.395257 14049 raft_consensus.cc:686] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:00:28.390604 14186 log.cc:924] Time spent T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260807 08:00:28.396134 14049 consensus_queue.cc:237] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.397329 13951 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.397989 13876 ts_tablet_manager.cc:1428] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.230s user 0.029s sys 0.035s
I20260807 08:00:28.403139 13876 tablet_bootstrap.cc:492] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:28.407068 13874 tablet_bootstrap.cc:654] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.413993 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:28.414748 13591 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:28.417845 13508 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 08:00:28.420575 13876 tablet_bootstrap.cc:654] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.422188 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:28.422879 13816 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.423095 13591 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.428757 13951 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.429291 13951 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.430420 13816 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.431866 13951 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.317935 13592 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.435855 13657 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.437800 14168 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.438308 13951 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.439116 13951 raft_consensus.cc:686] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:28.440598 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.446283 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.450452 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:28.440091 13951 consensus_queue.cc:237] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.461484 13951 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.462026 13951 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.465353 13951 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.467339 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:28.469503 14111 consensus_queue.cc:1035] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.470219 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:28.470567 13657 leader_election.cc:341] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.471252 14168 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 08:00:28.471504 13657 leader_election.cc:341] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:28.473856 13661 leader_election.cc:341] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.474361 13661 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:28.474543 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.475193 14168 raft_consensus.cc:2726] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:28.475217 13588 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.477049 13661 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:28.477870 14168 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.478257 14168 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.478595 14168 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.482076 13592 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
W20260807 08:00:28.477730 13769 leader_election.cc:341] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.485420 14049 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.485954 13875 tablet_bootstrap.cc:492] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:28.490170 13875 ts_tablet_manager.cc:1397] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.289s user 0.011s sys 0.002s
I20260807 08:00:28.488585 14168 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.494448 13971 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.497241 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:28.498126 13592 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:28.502000 13769 leader_election.cc:341] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.503296 13769 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:28.504918 13971 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:28.509017 13592 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.510635 13661 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.511888 13951 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.512544 13951 raft_consensus.cc:686] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:28.513746 13951 consensus_queue.cc:237] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.519790 14168 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.523180 13875 raft_consensus.cc:348] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.524246 13875 raft_consensus.cc:374] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.528208 13875 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.538847 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:28.542694 14168 consensus_queue.cc:1035] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.530239 13875 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:28.560386 13657 leader_election.cc:341] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.585040 13875 ts_tablet_manager.cc:1428] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.094s user 0.007s sys 0.002s
I20260807 08:00:28.586189 13874 tablet_bootstrap.cc:492] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:28.586642 13874 ts_tablet_manager.cc:1397] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.334s user 0.053s sys 0.067s
I20260807 08:00:28.588109 13876 tablet_bootstrap.cc:492] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:28.588580 13876 ts_tablet_manager.cc:1397] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.186s user 0.017s sys 0.020s
I20260807 08:00:28.589074 13874 raft_consensus.cc:348] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.589857 13874 raft_consensus.cc:374] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.590111 13874 raft_consensus.cc:729] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.590799 13874 consensus_queue.cc:260] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.592044 13876 raft_consensus.cc:348] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.592885 13874 ts_tablet_manager.cc:1428] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 08:00:28.592959 13876 raft_consensus.cc:374] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.593472 13876 raft_consensus.cc:729] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.593888 13874 tablet_bootstrap.cc:492] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:28.594413 13876 consensus_queue.cc:260] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.596572 13876 ts_tablet_manager.cc:1428] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260807 08:00:28.597616 13876 tablet_bootstrap.cc:492] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:28.599620 13874 tablet_bootstrap.cc:654] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.601292 13875 tablet_bootstrap.cc:492] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:28.612258 13876 tablet_bootstrap.cc:654] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.601792 14040 consensus_queue.cc:1035] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.614521 14040 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.616757 13874 tablet_bootstrap.cc:492] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:28.617194 13874 ts_tablet_manager.cc:1397] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.001s
I20260807 08:00:28.619774 13874 raft_consensus.cc:348] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.620457 13874 raft_consensus.cc:374] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.620728 13874 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.621459 13874 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.624282 13874 ts_tablet_manager.cc:1428] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260807 08:00:28.624471 13875 tablet_bootstrap.cc:654] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.625324 13874 tablet_bootstrap.cc:492] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:28.631111 13874 tablet_bootstrap.cc:654] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.631703 13875 tablet_bootstrap.cc:492] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:28.632185 13875 ts_tablet_manager.cc:1397] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.000s
I20260807 08:00:28.634806 13875 raft_consensus.cc:348] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.635547 13875 raft_consensus.cc:374] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.635051 13431 catalog_manager.cc:5495] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:28.635820 13875 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.640875 13875 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.643105 13875 ts_tablet_manager.cc:1428] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260807 08:00:28.644007 13875 tablet_bootstrap.cc:492] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:28.649113 13876 tablet_bootstrap.cc:492] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:28.649632 13876 ts_tablet_manager.cc:1397] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.002s
I20260807 08:00:28.652190 13876 raft_consensus.cc:348] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.653343 13876 raft_consensus.cc:374] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.653667 13876 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.654383 13876 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.655506 14166 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.658804 13874 tablet_bootstrap.cc:492] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:28.659247 13874 ts_tablet_manager.cc:1397] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.011s
I20260807 08:00:28.659559 13876 ts_tablet_manager.cc:1428] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260807 08:00:28.660483 13876 tablet_bootstrap.cc:492] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:28.661548 13874 raft_consensus.cc:348] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.662206 13874 raft_consensus.cc:374] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.662487 13874 raft_consensus.cc:729] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.663177 13874 consensus_queue.cc:260] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.665340 13874 ts_tablet_manager.cc:1428] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260807 08:00:28.666719 13875 tablet_bootstrap.cc:654] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.666878 13876 tablet_bootstrap.cc:654] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.671221 13874 tablet_bootstrap.cc:492] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:28.680092 13430 catalog_manager.cc:5495] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:28.689174 13874 tablet_bootstrap.cc:654] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.682425 13430 catalog_manager.cc:5495] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:28.691057 13876 tablet_bootstrap.cc:492] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:28.691501 13876 ts_tablet_manager.cc:1397] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.004s
I20260807 08:00:28.693135 13875 tablet_bootstrap.cc:492] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:28.693635 13875 ts_tablet_manager.cc:1397] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.006s
I20260807 08:00:28.694422 13876 raft_consensus.cc:348] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.695761 13876 raft_consensus.cc:374] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.696017 13876 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.696738 13875 raft_consensus.cc:348] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.696640 13876 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.697489 13875 raft_consensus.cc:374] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.697839 13875 raft_consensus.cc:729] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.698608 13875 consensus_queue.cc:260] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.699718 13876 ts_tablet_manager.cc:1428] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260807 08:00:28.700574 13876 tablet_bootstrap.cc:492] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:28.700697 13875 ts_tablet_manager.cc:1428] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260807 08:00:28.701619 13875 tablet_bootstrap.cc:492] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:28.705783 13876 tablet_bootstrap.cc:654] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.707720 13875 tablet_bootstrap.cc:654] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.713371 14166 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.713922 14166 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.716825 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:00:28.718309 13661 leader_election.cc:341] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.718333 14166 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.719316 13874 tablet_bootstrap.cc:492] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:28.719858 13874 ts_tablet_manager.cc:1397] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.027s
I20260807 08:00:28.720432 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:28.721093 13817 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.722052 13876 tablet_bootstrap.cc:492] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:28.722520 13657 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:28.723348 14168 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.723695 14168 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.724033 14168 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.722524 13876 ts_tablet_manager.cc:1397] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20260807 08:00:28.724498 13874 raft_consensus.cc:348] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.725459 13874 raft_consensus.cc:374] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.725800 13874 raft_consensus.cc:729] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.726605 13874 consensus_queue.cc:260] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.727111 13876 raft_consensus.cc:348] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.727842 13876 raft_consensus.cc:374] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.728087 13876 raft_consensus.cc:729] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.728681 13876 consensus_queue.cc:260] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.729550 13874 ts_tablet_manager.cc:1428] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260807 08:00:28.730422 13874 tablet_bootstrap.cc:492] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:28.730954 13876 ts_tablet_manager.cc:1428] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 08:00:28.731827 13876 tablet_bootstrap.cc:492] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:28.736423 13874 tablet_bootstrap.cc:654] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.736819 13876 tablet_bootstrap.cc:654] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.738021 14168 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.742076 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:28.742935 13816 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.747011 13884 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.747623 13884 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.752688 13816 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.760370 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:28.761904 14185 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.762619 14185 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.771322 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:28.772276 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.765389 14185 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:28.775810 14166 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.776288 14166 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.773128 13592 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:28.778290 14166 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.772316 13711 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:28.782001 14168 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.782509 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:28.784000 13657 leader_election.cc:341] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.784725 13657 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.785977 14166 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.786453 14166 raft_consensus.cc:686] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:00:28.786901 13661 leader_election.cc:341] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.787302 14166 consensus_queue.cc:237] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.786345 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.789276 14040 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.789955 14040 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.791453 13971 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.791567 13875 tablet_bootstrap.cc:492] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:28.792179 13875 ts_tablet_manager.cc:1397] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.001s
I20260807 08:00:28.791934 13971 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.792086 14040 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:00:28.793375 13661 leader_election.cc:341] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.793555 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:28.793917 13661 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:28.794241 13816 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.794880 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.794059 13971 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:28.778445 13769 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:28.795289 14040 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:28.795487 13591 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:28.797032 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:28.797245 13971 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.797668 13971 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.797705 13711 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:28.798096 13971 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.799103 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.799103 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.799948 13588 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:28.800668 13963 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.801033 13963 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.801133 13661 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.795403 13884 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.801393 13963 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.802546 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:28.803308 13817 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:28.803467 14105 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.803869 14105 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.804250 14105 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.808384 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:28.809006 13711 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:28.812563 14105 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.818501 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:28.819077 13817 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.811904 13769 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:28.820626 14185 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:28.821033 14185 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:28.821376 14185 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.808930 13875 raft_consensus.cc:348] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.822675 13875 raft_consensus.cc:374] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.822973 13875 raft_consensus.cc:729] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.822048 14105 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.824309 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:28.827013 13874 tablet_bootstrap.cc:492] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:28.827280 13875 consensus_queue.cc:260] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.832607 13874 ts_tablet_manager.cc:1397] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.102s user 0.041s sys 0.018s
I20260807 08:00:28.831579 13430 catalog_manager.cc:5495] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:28.835772 13874 raft_consensus.cc:348] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.836666 13874 raft_consensus.cc:374] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.836982 13874 raft_consensus.cc:729] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.839876 13817 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:28.841423 13657 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.809697 13963 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.842579 14105 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.844026 14105 raft_consensus.cc:686] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:28.844785 13591 raft_consensus.cc:2379] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 1: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:00:28.845184 14168 consensus_queue.cc:1035] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.845126 14105 consensus_queue.cc:237] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.842885 13875 ts_tablet_manager.cc:1428] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.050s user 0.003s sys 0.003s
I20260807 08:00:28.838457 13874 consensus_queue.cc:260] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.830626 14185 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.849578 13875 tablet_bootstrap.cc:492] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:28.848335 13971 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.856820 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:28.860175 13874 ts_tablet_manager.cc:1428] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.027s user 0.004s sys 0.003s
I20260807 08:00:28.860934 13874 tablet_bootstrap.cc:492] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:28.866554 13963 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.867549 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:28.868628 13816 raft_consensus.cc:2379] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 1: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:00:28.870605 13537 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:28.871388 14113 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 08:00:28.872102 14049 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.881326 13971 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:28.881984 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:28.882652 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:28.882818 13592 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.883261 13710 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.884099 13875 tablet_bootstrap.cc:654] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.880889 13431 catalog_manager.cc:5495] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:28.886965 13431 catalog_manager.cc:5495] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:28.885758 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:28.889109 13588 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.894744 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:28.895531 13705 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.898706 13876 tablet_bootstrap.cc:492] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:28.899241 13876 ts_tablet_manager.cc:1397] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.168s user 0.034s sys 0.044s
I20260807 08:00:28.903061 13592 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:28.897991 14185 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:28.910872 14105 consensus_queue.cc:1035] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.908406 13710 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:28.906673 13769 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:28.917384 13876 raft_consensus.cc:348] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.917935 13705 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:28.918285 13876 raft_consensus.cc:374] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:28.918668 13876 raft_consensus.cc:729] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:28.920454 13769 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:28.921379 14113 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.922668 13876 consensus_queue.cc:260] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.914880 14185 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.922322 13971 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:28.922024 14113 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.924682 13971 raft_consensus.cc:686] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:28.925546 13971 consensus_queue.cc:237] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.926205 14113 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:28.927641 14185 raft_consensus.cc:686] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:28.928781 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:28.928529 14185 consensus_queue.cc:237] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.929980 14105 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:28.932701 13876 ts_tablet_manager.cc:1428] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20260807 08:00:28.933604 13876 tablet_bootstrap.cc:492] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
W20260807 08:00:28.932256 13537 leader_election.cc:341] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.935824 13588 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:28.936388 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:00:28.938273 13540 leader_election.cc:341] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:28.938884 13540 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:28.942060 14049 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.945051 14234 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:28.945710 14234 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:28.947513 14105 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:28.948771 14234 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:28.950909 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:28.951622 14113 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:28.953150 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:28.954242 13589 raft_consensus.cc:2379] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 1: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:00:28.958076 13769 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:28.965911 13874 tablet_bootstrap.cc:654] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:28.965296 14049 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:28.973104 13876 tablet_bootstrap.cc:654] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.028657 14049 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:29.050509 13875 tablet_bootstrap.cc:492] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:29.051141 13875 ts_tablet_manager.cc:1397] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.202s user 0.021s sys 0.019s
I20260807 08:00:29.085196 14166 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260807 08:00:29.093323 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 135333us
I20260807 08:00:29.097640 13874 tablet_bootstrap.cc:492] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:29.098119 13874 ts_tablet_manager.cc:1397] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.237s user 0.010s sys 0.016s
I20260807 08:00:29.101904 13430 catalog_manager.cc:5495] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:29.104065 13430 catalog_manager.cc:5495] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:29.101284 14246 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.119690 13874 raft_consensus.cc:348] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.120862 13874 raft_consensus.cc:374] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.121294 13874 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.123668 13874 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.129164 13874 ts_tablet_manager.cc:1428] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.031s user 0.007s sys 0.001s
I20260807 08:00:29.130466 13874 tablet_bootstrap.cc:492] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:29.140328 14185 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.123701 14246 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.157938 14185 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.161808 14246 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:29.162348 14185 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:29.200153 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:29.230443 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:29.239370 13876 tablet_bootstrap.cc:492] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:29.239903 13876 ts_tablet_manager.cc:1397] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.306s user 0.014s sys 0.002s
I20260807 08:00:29.242533 13876 raft_consensus.cc:348] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.243280 13876 raft_consensus.cc:374] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.243548 13876 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.244226 13876 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.249509 13876 ts_tablet_manager.cc:1428] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.009s user 0.001s sys 0.007s
W20260807 08:00:29.251564 13769 leader_election.cc:341] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.253197 13876 tablet_bootstrap.cc:492] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:29.254760 13875 raft_consensus.cc:348] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.255277 13875 raft_consensus.cc:374] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.255518 13875 raft_consensus.cc:729] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
W20260807 08:00:29.257016 13769 leader_election.cc:341] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.256970 13875 consensus_queue.cc:260] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.259716 13875 ts_tablet_manager.cc:1428] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.208s user 0.008s sys 0.000s
I20260807 08:00:29.265847 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:29.266808 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:29.273144 14168 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 08:00:29.275293 13769 leader_election.cc:341] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.275820 13769 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:29.275512 13875 tablet_bootstrap.cc:492] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:29.277417 13874 tablet_bootstrap.cc:654] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.278282 14253 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 08:00:29.288648 13769 leader_election.cc:341] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.289050 13769 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:29.291144 14049 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:29.299079 14040 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260807 08:00:29.301082 14255 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.301903 14255 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.304455 14255 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:29.309096 13590 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.309288 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:29.312171 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:29.312803 13711 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
W20260807 08:00:29.313428 13769 leader_election.cc:341] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.313994 13875 tablet_bootstrap.cc:654] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.315579 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:29.318387 14253 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:29.318742 14253 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:29.319075 14253 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.320881 14168 consensus_queue.cc:1035] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:29.328660 13876 tablet_bootstrap.cc:654] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.332945 14113 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.333877 14113 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.337955 14113 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.340991 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:29.341751 13711 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:29.342883 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:29.344488 14113 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:29.344863 14113 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:29.349756 14113 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.353312 14253 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.358151 14113 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.359243 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:00:29.360975 13769 leader_election.cc:341] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.361550 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:29.362110 13711 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.362821 14253 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:29.365052 14255 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:29.368302 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:29.368865 13709 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.369401 13876 tablet_bootstrap.cc:492] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:29.369982 14113 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.370579 13876 ts_tablet_manager.cc:1397] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.120s user 0.009s sys 0.007s
I20260807 08:00:29.374126 13711 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:29.375598 13709 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:29.375655 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:29.376781 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:29.377197 13874 tablet_bootstrap.cc:492] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:29.378433 14255 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:29.378562 13884 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:29.378940 14255 raft_consensus.cc:686] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:29.379084 13884 raft_consensus.cc:686] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:29.379776 14255 consensus_queue.cc:237] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.380344 13884 consensus_queue.cc:237] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.385195 13874 ts_tablet_manager.cc:1397] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.255s user 0.025s sys 0.012s
I20260807 08:00:29.385887 13875 tablet_bootstrap.cc:492] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:29.388098 13875 ts_tablet_manager.cc:1397] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.113s user 0.007s sys 0.009s
I20260807 08:00:29.390322 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:29.390586 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:29.373275 13876 raft_consensus.cc:348] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.395720 13876 raft_consensus.cc:374] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.404595 13876 raft_consensus.cc:729] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.398573 13874 raft_consensus.cc:348] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.404757 13875 raft_consensus.cc:348] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.405236 13874 raft_consensus.cc:374] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.405431 13875 raft_consensus.cc:374] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.405558 13874 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.405798 13875 raft_consensus.cc:729] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.405622 14105 consensus_queue.cc:1035] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:29.403409 14040 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:29.406432 13874 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.408535 14253 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:29.409209 13876 consensus_queue.cc:260] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:29.411269 13537 leader_election.cc:341] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.412031 13875 consensus_queue.cc:260] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:29.412465 13537 leader_election.cc:341] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.420123 13875 ts_tablet_manager.cc:1428] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.032s user 0.004s sys 0.003s
I20260807 08:00:29.421202 13875 tablet_bootstrap.cc:492] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:29.435482 13874 ts_tablet_manager.cc:1428] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20260807 08:00:29.436266 13874 tablet_bootstrap.cc:492] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:29.436998 13876 ts_tablet_manager.cc:1428] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.066s user 0.006s sys 0.000s
I20260807 08:00:29.437731 13876 tablet_bootstrap.cc:492] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:29.442515 13876 tablet_bootstrap.cc:654] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.453322 13431 catalog_manager.cc:5495] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:29.463203 14168 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.463691 14168 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.464213 14253 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:29.465647 14168 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.467037 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:29.467077 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:29.467896 13591 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:29.469030 13661 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:29.470036 14040 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:29.469399 13430 catalog_manager.cc:5495] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
W20260807 08:00:29.470177 13657 leader_election.cc:341] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.470435 14040 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:29.470808 14040 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.481309 14040 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.488405 14040 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.493404 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:29.494046 13591 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.497999 13874 tablet_bootstrap.cc:654] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.507889 13875 tablet_bootstrap.cc:654] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.513085 13591 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:29.514333 13661 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:29.516218 14040 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:29.516992 14040 raft_consensus.cc:686] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:29.508456 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:29.518468 14040 consensus_queue.cc:237] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:29.519277 13657 leader_election.cc:341] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.527967 14253 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260807 08:00:29.556509 13874 tablet_bootstrap.cc:492] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:29.556975 13874 ts_tablet_manager.cc:1397] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.121s user 0.024s sys 0.010s
I20260807 08:00:29.561441 13874 raft_consensus.cc:348] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.572284 13876 tablet_bootstrap.cc:492] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:29.572783 13876 ts_tablet_manager.cc:1397] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.135s user 0.014s sys 0.011s
I20260807 08:00:29.574816 13431 catalog_manager.cc:5495] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:29.575219 13876 raft_consensus.cc:348] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.575968 13876 raft_consensus.cc:374] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.576238 13876 raft_consensus.cc:729] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.562153 13874 raft_consensus.cc:374] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.598798 13874 raft_consensus.cc:729] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.600227 13874 consensus_queue.cc:260] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.604334 13874 ts_tablet_manager.cc:1428] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20260807 08:00:29.608443 13875 tablet_bootstrap.cc:492] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:29.608916 13875 ts_tablet_manager.cc:1397] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.188s user 0.020s sys 0.005s
I20260807 08:00:29.576859 13876 consensus_queue.cc:260] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.621469 13876 ts_tablet_manager.cc:1428] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20260807 08:00:29.622437 13876 tablet_bootstrap.cc:492] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:29.623208 13875 raft_consensus.cc:348] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.623946 13875 raft_consensus.cc:374] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.624212 13875 raft_consensus.cc:729] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.624841 13875 consensus_queue.cc:260] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.626070 13874 tablet_bootstrap.cc:492] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:29.632328 13876 tablet_bootstrap.cc:654] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.654695 13875 ts_tablet_manager.cc:1428] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.045s user 0.003s sys 0.003s
W20260807 08:00:29.659286 13508 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 08:00:29.662225 14040 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.662688 14040 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.664706 14040 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.665611 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:29.666960 13884 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:00:29.666976 13657 leader_election.cc:341] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.667647 13874 tablet_bootstrap.cc:654] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.667564 13884 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.669473 13875 tablet_bootstrap.cc:492] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:29.669764 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:29.671170 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:29.671779 13705 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:29.673028 13540 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:29.673800 13876 tablet_bootstrap.cc:492] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:29.673938 14113 raft_consensus.cc:2781] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:29.674252 13876 ts_tablet_manager.cc:1397] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.001s
W20260807 08:00:29.671793 13661 leader_election.cc:341] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.674741 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:29.674273 14113 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:29.676029 14113 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.677119 13876 raft_consensus.cc:348] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.677803 13876 raft_consensus.cc:374] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.678050 13876 raft_consensus.cc:729] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.682044 14040 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:29.682276 13875 tablet_bootstrap.cc:654] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.683216 13876 consensus_queue.cc:260] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.686721 13876 ts_tablet_manager.cc:1428] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260807 08:00:29.688853 13876 tablet_bootstrap.cc:492] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:29.692360 14113 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.694613 14113 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.696380 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:29.696960 13705 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.697062 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:29.697907 13812 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.698009 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:29.695871 13884 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.707746 13812 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:29.710541 13537 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:29.707998 13705 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:29.709576 13876 tablet_bootstrap.cc:654] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.712227 13884 raft_consensus.cc:2781] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:29.713855 13884 raft_consensus.cc:686] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:29.714638 13884 consensus_queue.cc:237] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.718600 13875 tablet_bootstrap.cc:492] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:29.719691 13875 ts_tablet_manager.cc:1397] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.003s
I20260807 08:00:29.711999 13874 tablet_bootstrap.cc:492] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:29.721998 13874 ts_tablet_manager.cc:1397] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.011s
I20260807 08:00:29.724375 13875 raft_consensus.cc:348] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.725189 13875 raft_consensus.cc:374] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.725540 13875 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.726296 13875 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.728123 13876 tablet_bootstrap.cc:492] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:29.728538 13876 ts_tablet_manager.cc:1397] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.005s
I20260807 08:00:29.728614 13875 ts_tablet_manager.cc:1428] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260807 08:00:29.730237 13875 tablet_bootstrap.cc:492] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:29.729686 13431 catalog_manager.cc:5495] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:29.731388 13876 raft_consensus.cc:348] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.732031 13876 raft_consensus.cc:374] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.732270 13876 raft_consensus.cc:729] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.732825 13876 consensus_queue.cc:260] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.734385 13874 raft_consensus.cc:348] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.735041 13874 raft_consensus.cc:374] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.735282 13874 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.735937 13874 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.737957 13884 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.738386 13884 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.740195 13884 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.741856 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:29.742146 13876 ts_tablet_manager.cc:1428] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20260807 08:00:29.743484 13876 tablet_bootstrap.cc:492] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:29.744784 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:29.745586 13705 raft_consensus.cc:2379] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 1: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:00:29.751927 13874 ts_tablet_manager.cc:1428] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.030s user 0.004s sys 0.002s
I20260807 08:00:29.752789 13874 tablet_bootstrap.cc:492] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:29.753029 13875 tablet_bootstrap.cc:654] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.757815 13537 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:29.758570 13884 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.764747 13884 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:29.766855 13876 tablet_bootstrap.cc:654] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.769695 13874 tablet_bootstrap.cc:654] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.775388 13884 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.776821 13884 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.779855 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:29.780755 13705 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:29.781841 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:29.782702 14113 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:29.783061 14113 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:29.783389 14113 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.785940 13874 tablet_bootstrap.cc:492] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:29.786355 13874 ts_tablet_manager.cc:1397] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.003s
I20260807 08:00:29.789149 13874 raft_consensus.cc:348] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.789855 13874 raft_consensus.cc:374] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.790117 13874 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.790707 13874 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.793321 13876 tablet_bootstrap.cc:492] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:29.793682 13874 ts_tablet_manager.cc:1428] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 08:00:29.793789 13876 ts_tablet_manager.cc:1397] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.004s
I20260807 08:00:29.795022 13874 tablet_bootstrap.cc:492] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:29.797010 13876 raft_consensus.cc:348] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.797361 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:29.797785 13876 raft_consensus.cc:374] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.798154 13876 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.798763 13876 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:29.799487 13537 leader_election.cc:341] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.800452 13874 tablet_bootstrap.cc:654] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.800540 13884 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.800977 13876 ts_tablet_manager.cc:1428] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260807 08:00:29.811539 13874 tablet_bootstrap.cc:492] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:29.811992 13874 ts_tablet_manager.cc:1397] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.012s
I20260807 08:00:29.812762 13876 tablet_bootstrap.cc:492] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:29.818552 13875 tablet_bootstrap.cc:492] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:29.822346 13875 ts_tablet_manager.cc:1397] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.092s user 0.022s sys 0.001s
I20260807 08:00:29.814889 13874 raft_consensus.cc:348] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.823395 13874 raft_consensus.cc:374] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.823720 13874 raft_consensus.cc:729] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.824791 13876 tablet_bootstrap.cc:654] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.824458 13874 consensus_queue.cc:260] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.824875 13875 raft_consensus.cc:348] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.815263 14113 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.827608 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:29.827966 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:29.828364 13705 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:29.829267 13537 leader_election.cc:341] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.829873 14113 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:29.830417 13875 raft_consensus.cc:374] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.830899 13875 raft_consensus.cc:729] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.831627 13875 consensus_queue.cc:260] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.833855 13874 ts_tablet_manager.cc:1428] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.022s user 0.007s sys 0.002s
I20260807 08:00:29.834779 13874 tablet_bootstrap.cc:492] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:29.835320 13875 ts_tablet_manager.cc:1428] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260807 08:00:29.838168 13705 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:29.838649 14255 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.839110 14255 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.839522 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:29.840891 14113 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:29.840927 14255 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:29.841831 13875 tablet_bootstrap.cc:492] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:29.842245 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:29.843017 14113 raft_consensus.cc:686] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:29.845495 13874 tablet_bootstrap.cc:654] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.846990 14113 consensus_queue.cc:237] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.854259 14253 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.854916 13875 tablet_bootstrap.cc:654] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.854753 14253 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.856592 14253 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:29.858413 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:00:29.859803 13769 leader_election.cc:341] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.864801 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:29.865621 13590 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 1: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:00:29.867869 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:29.868482 13590 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:29.870311 13769 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:29.870608 13875 tablet_bootstrap.cc:492] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:29.871194 13875 ts_tablet_manager.cc:1397] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.000s
I20260807 08:00:29.873633 14246 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.875753 13874 tablet_bootstrap.cc:492] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:29.876211 13874 ts_tablet_manager.cc:1397] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.017s
I20260807 08:00:29.878110 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:29.882076 14255 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:29.882470 14255 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:29.882818 14255 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.887225 13875 raft_consensus.cc:348] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.888132 13875 raft_consensus.cc:374] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.888412 13875 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.889002 13875 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.892997 13428 catalog_manager.cc:5495] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:29.897703 14246 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:29.906126 13875 ts_tablet_manager.cc:1428] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260807 08:00:29.907228 13874 raft_consensus.cc:348] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.908185 13874 raft_consensus.cc:374] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.908510 13874 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.909271 13874 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.910048 13875 tablet_bootstrap.cc:492] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:29.916445 13874 ts_tablet_manager.cc:1428] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.040s user 0.007s sys 0.002s
I20260807 08:00:29.917363 13874 tablet_bootstrap.cc:492] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:29.925298 13876 tablet_bootstrap.cc:492] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:29.925853 13876 ts_tablet_manager.cc:1397] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.113s user 0.015s sys 0.017s
I20260807 08:00:29.929899 13875 tablet_bootstrap.cc:654] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:29.927029 14255 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.941251 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:29.933085 13876 raft_consensus.cc:348] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.940438 14255 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:29.941908 13876 raft_consensus.cc:374] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:29.942315 13876 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:29.942611 13874 tablet_bootstrap.cc:654] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
W20260807 08:00:29.942761 13769 leader_election.cc:341] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:29.940626 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:29.944169 13590 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.945120 13812 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:29.953105 14255 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:29.953729 14113 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:29.959020 13590 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:29.960186 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:29.961318 14255 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:29.962847 14255 raft_consensus.cc:686] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:29.964037 14255 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.982120 14168 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:29.982527 13884 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:29.985030 13876 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.987676 14113 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:29.988119 14113 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:29.991812 14246 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:29.999305 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:30.000142 13817 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:30.001617 13537 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:30.002633 13884 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:30.002987 13884 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:30.002913 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:30.003419 13884 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:30.004070 13540 leader_election.cc:341] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.004968 14113 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:30.010686 13884 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.013477 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:00:30.014585 13540 leader_election.cc:341] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.015322 13884 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:30.020463 14246 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:30.020973 14168 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:30.021581 14246 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.025370 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:30.025940 13817 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:30.030973 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:00:30.033311 13769 leader_election.cc:341] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.035033 14246 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:30.035614 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:30.036425 13711 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:30.038009 13769 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:30.047868 14246 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:30.049563 14246 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:30.051074 14246 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:30.065819 14168 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:30.066480 14168 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.069599 14168 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:30.070312 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:30.070583 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:30.071399 13814 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:30.072255 13592 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:30.072402 13657 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:30.079074 14168 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:30.079020 14246 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.079634 14168 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:30.080116 14168 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:30.081048 14246 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:30.083158 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:00:30.085153 13769 leader_election.cc:341] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.150228 13430 catalog_manager.cc:5495] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:30.164296 13876 ts_tablet_manager.cc:1428] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.238s user 0.013s sys 0.003s
I20260807 08:00:30.165294 13876 tablet_bootstrap.cc:492] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:30.188308 14105 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:30.188835 14105 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.175981 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:30.176055 13817 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:30.216289 13705 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:30.242846 13537 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:30.280471 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:30.252418 13884 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:30.286058 13884 raft_consensus.cc:686] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:30.286911 13884 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.291254 14105 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:00:30.292073 13661 leader_election.cc:341] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.306311 13876 tablet_bootstrap.cc:654] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:30.308822 13874 tablet_bootstrap.cc:492] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:30.309423 13874 ts_tablet_manager.cc:1397] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.392s user 0.072s sys 0.134s
I20260807 08:00:30.335371 13705 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:30.359594 13875 tablet_bootstrap.cc:492] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:30.360021 13875 ts_tablet_manager.cc:1397] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.453s user 0.073s sys 0.063s
I20260807 08:00:30.334406 13963 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260807 08:00:30.357641 14336 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:30.380295 14168 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.383638 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:30.384287 13592 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:30.388794 14168 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:30.393601 13769 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:30.397953 14049 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:30.421245 13874 raft_consensus.cc:348] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.422231 13874 raft_consensus.cc:374] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:30.422593 13874 raft_consensus.cc:729] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:30.391718 14336 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.424427 13874 consensus_queue.cc:260] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.420249 14049 raft_consensus.cc:686] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:30.443131 13963 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260807 08:00:30.448475 13875 raft_consensus.cc:348] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.449173 13875 raft_consensus.cc:374] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:30.441823 14351 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260807 08:00:30.453096 13592 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:30.449434 13875 raft_consensus.cc:729] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:30.455422 13875 consensus_queue.cc:260] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.462095 14360 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:30.463980 14360 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.466037 14360 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:30.469485 13874 ts_tablet_manager.cc:1428] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.160s user 0.017s sys 0.003s
I20260807 08:00:30.470502 13874 tablet_bootstrap.cc:492] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:30.472314 13875 ts_tablet_manager.cc:1428] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.112s user 0.005s sys 0.003s
I20260807 08:00:30.473178 13875 tablet_bootstrap.cc:492] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:30.469808 13661 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:30.467116 14336 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:30.468290 14049 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.476500 13876 tablet_bootstrap.cc:492] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:30.582198 13876 ts_tablet_manager.cc:1397] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.417s user 0.014s sys 0.016s
I20260807 08:00:30.585911 13875 tablet_bootstrap.cc:654] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:30.586673 13874 tablet_bootstrap.cc:654] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:30.586894 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:30.481720 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:30.479871 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:30.588196 13706 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:30.590415 13876 raft_consensus.cc:348] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.590950 13876 raft_consensus.cc:374] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:30.591166 13876 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:30.591714 13876 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.562438 13431 catalog_manager.cc:5495] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:30.481170 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:30.511479 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:30.597146 13816 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:30.598865 13769 leader_election.cc:341] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.601810 13876 ts_tablet_manager.cc:1428] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
W20260807 08:00:30.602248 13769 leader_election.cc:341] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.602838 13876 tablet_bootstrap.cc:492] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:30.602921 13769 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:30.605075 14339 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:30.605417 14339 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:30.605799 14339 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:30.479140 14331 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:30.480600 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:30.638214 13769 consensus_peers.cc:487] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Couldn't send request to peer 9210dbc4cc9c45e5b26a463b705d362c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260807 08:00:30.640159 13769 leader_election.cc:341] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.640553 13769 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:30.644945 14255 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:30.650686 13874 tablet_bootstrap.cc:492] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:30.638419 14331 raft_consensus.cc:686] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:30.656469 13874 ts_tablet_manager.cc:1397] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.186s user 0.029s sys 0.020s
I20260807 08:00:30.660887 13876 tablet_bootstrap.cc:654] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:30.665097 14331 consensus_queue.cc:237] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.655162 14339 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.674081 14350 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:30.676592 14361 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260807 08:00:30.676608 14350 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.687888 14350 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:30.699098 13816 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:30.711974 13875 tablet_bootstrap.cc:492] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:30.712636 13875 ts_tablet_manager.cc:1397] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.240s user 0.009s sys 0.005s
I20260807 08:00:30.717979 13811 raft_consensus.cc:2379] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 1: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:00:30.718838 14339 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:30.720885 13875 raft_consensus.cc:348] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.721848 13875 raft_consensus.cc:374] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:30.722162 13875 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:30.731117 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:30.731973 13587 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:30.733876 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:30.802959 13706 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:30.802698 13875 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.804844 13875 ts_tablet_manager.cc:1428] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.092s user 0.004s sys 0.004s
I20260807 08:00:30.808009 13875 tablet_bootstrap.cc:492] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:30.810108 13428 catalog_manager.cc:5495] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
W20260807 08:00:30.812091 13508 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 08:00:30.823540 13874 raft_consensus.cc:348] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.824268 13874 raft_consensus.cc:374] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:30.824772 13874 raft_consensus.cc:729] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:30.825501 13874 consensus_queue.cc:260] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.833341 13708 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:30.789163 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:30.848438 13876 tablet_bootstrap.cc:492] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:30.848877 13876 ts_tablet_manager.cc:1397] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.246s user 0.018s sys 0.014s
I20260807 08:00:30.850158 14351 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:30.860562 13876 raft_consensus.cc:348] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.861280 13876 raft_consensus.cc:374] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:30.861577 13876 raft_consensus.cc:729] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:30.862116 13876 consensus_queue.cc:260] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.865001 13874 ts_tablet_manager.cc:1428] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.208s user 0.004s sys 0.002s
I20260807 08:00:30.866449 13874 tablet_bootstrap.cc:492] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:30.870796 13876 ts_tablet_manager.cc:1428] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.022s user 0.006s sys 0.002s
I20260807 08:00:30.869264 13430 catalog_manager.cc:5495] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:30.871681 13876 tablet_bootstrap.cc:492] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:30.898180 13875 tablet_bootstrap.cc:654] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:30.896346 13587 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
W20260807 08:00:30.900852 13537 consensus_peers.cc:487] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad -> Peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Couldn't send request to peer 7a889001d6494fe5b7cbbe7da1dae302. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:30.903134 13657 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:30.905871 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:30.906700 13814 raft_consensus.cc:2379] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 1: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:00:30.909616 13769 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:30.910594 14255 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:30.911067 14255 raft_consensus.cc:686] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:30.912184 14255 consensus_queue.cc:237] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.913540 13537 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:30.915150 14351 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:30.921486 14105 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:30.921972 14105 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:30.917192 14330 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260807 08:00:30.923910 14105 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:30.926543 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:30.928483 13876 tablet_bootstrap.cc:654] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
W20260807 08:00:30.939306 13657 leader_election.cc:341] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.917351 13706 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:30.946017 14346 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:30.984853 13876 tablet_bootstrap.cc:492] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:30.985273 13876 ts_tablet_manager.cc:1397] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.002s
I20260807 08:00:30.996405 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:00:30.998309 13661 leader_election.cc:341] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:30.998713 13661 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:30.999392 14340 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:31.000809 13876 raft_consensus.cc:348] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.007609 13876 raft_consensus.cc:374] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.005079 13431 catalog_manager.cc:5495] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:31.008795 13876 raft_consensus.cc:729] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.015697 13876 consensus_queue.cc:260] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.017843 13876 ts_tablet_manager.cc:1428] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.032s user 0.008s sys 0.000s
I20260807 08:00:31.018673 13876 tablet_bootstrap.cc:492] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:31.020323 13875 tablet_bootstrap.cc:492] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:31.020821 13875 ts_tablet_manager.cc:1397] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.213s user 0.024s sys 0.016s
I20260807 08:00:31.022545 13874 tablet_bootstrap.cc:654] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.023471 13875 raft_consensus.cc:348] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.024282 13875 raft_consensus.cc:374] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.024695 13875 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.025952 13875 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.033828 13807 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.040491 13876 tablet_bootstrap.cc:654] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.046344 13874 tablet_bootstrap.cc:492] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:31.046783 13874 ts_tablet_manager.cc:1397] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.181s user 0.011s sys 0.000s
I20260807 08:00:31.046900 13875 ts_tablet_manager.cc:1428] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.026s user 0.001s sys 0.006s
I20260807 08:00:31.054492 13874 raft_consensus.cc:348] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.055209 13874 raft_consensus.cc:374] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.055768 13874 raft_consensus.cc:729] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.056339 13874 consensus_queue.cc:260] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.064679 13874 ts_tablet_manager.cc:1428] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260807 08:00:31.067028 13875 tablet_bootstrap.cc:492] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:31.070798 13874 tablet_bootstrap.cc:492] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:31.074795 13876 tablet_bootstrap.cc:492] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:31.075461 13876 ts_tablet_manager.cc:1397] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.003s
I20260807 08:00:31.076295 13875 tablet_bootstrap.cc:654] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.078294 13876 raft_consensus.cc:348] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.079033 13876 raft_consensus.cc:374] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.079291 13876 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.080227 13874 tablet_bootstrap.cc:654] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.079991 13876 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.088491 13876 ts_tablet_manager.cc:1428] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260807 08:00:31.089427 13876 tablet_bootstrap.cc:492] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:31.090711 13875 tablet_bootstrap.cc:492] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:31.090757 14105 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:31.091233 13875 ts_tablet_manager.cc:1397] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.004s
I20260807 08:00:31.093616 13875 raft_consensus.cc:348] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.094264 13875 raft_consensus.cc:374] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.094532 13875 raft_consensus.cc:729] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.095479 13876 tablet_bootstrap.cc:654] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.095176 13875 consensus_queue.cc:260] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.101738 13874 tablet_bootstrap.cc:492] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:31.102236 13874 ts_tablet_manager.cc:1397] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.001s
I20260807 08:00:31.104624 13874 raft_consensus.cc:348] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.105279 13874 raft_consensus.cc:374] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.105566 13874 raft_consensus.cc:729] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.108618 13875 ts_tablet_manager.cc:1428] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260807 08:00:31.109483 13875 tablet_bootstrap.cc:492] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:31.108944 13874 consensus_queue.cc:260] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.118778 13874 ts_tablet_manager.cc:1428] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260807 08:00:31.124553 13874 tablet_bootstrap.cc:492] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:31.126619 13875 tablet_bootstrap.cc:654] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.127107 13876 tablet_bootstrap.cc:492] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:31.127583 13876 ts_tablet_manager.cc:1397] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.010s
I20260807 08:00:31.130066 13874 tablet_bootstrap.cc:654] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.129952 13876 raft_consensus.cc:348] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.130630 13876 raft_consensus.cc:374] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.130900 13876 raft_consensus.cc:729] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.136251 13876 consensus_queue.cc:260] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.138454 13875 tablet_bootstrap.cc:492] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:31.138904 13875 ts_tablet_manager.cc:1397] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.010s
I20260807 08:00:31.139243 13876 ts_tablet_manager.cc:1428] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260807 08:00:31.140249 13876 tablet_bootstrap.cc:492] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:31.141295 13875 raft_consensus.cc:348] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.141969 13875 raft_consensus.cc:374] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.142225 13875 raft_consensus.cc:729] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.142896 13875 consensus_queue.cc:260] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.145402 13875 ts_tablet_manager.cc:1428] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260807 08:00:31.146255 13875 tablet_bootstrap.cc:492] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:31.146577 13876 tablet_bootstrap.cc:654] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.151361 13875 tablet_bootstrap.cc:654] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.153921 13874 tablet_bootstrap.cc:492] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:31.154383 13874 ts_tablet_manager.cc:1397] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.000s
I20260807 08:00:31.155721 13876 tablet_bootstrap.cc:492] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:31.156144 13876 ts_tablet_manager.cc:1397] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.002s
I20260807 08:00:31.156874 13874 raft_consensus.cc:348] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.157573 13874 raft_consensus.cc:374] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.157814 13874 raft_consensus.cc:729] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.158438 13876 raft_consensus.cc:348] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.159184 13876 raft_consensus.cc:374] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.159444 13876 raft_consensus.cc:729] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.160012 13876 consensus_queue.cc:260] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.158460 13874 consensus_queue.cc:260] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.162820 13876 ts_tablet_manager.cc:1428] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 08:00:31.163668 13876 tablet_bootstrap.cc:492] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:31.163978 13874 ts_tablet_manager.cc:1428] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260807 08:00:31.163867 14339 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:31.164997 13874 tablet_bootstrap.cc:492] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:31.167500 14346 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:31.168753 13876 tablet_bootstrap.cc:654] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.170564 13874 tablet_bootstrap.cc:654] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.192044 13874 tablet_bootstrap.cc:492] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:31.192613 13874 ts_tablet_manager.cc:1397] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20260807 08:00:31.193683 14330 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.194222 14330 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.196185 13874 raft_consensus.cc:348] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.196858 14330 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.197113 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:31.196861 13874 raft_consensus.cc:374] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.197877 13807 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:31.198125 13874 raft_consensus.cc:729] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.199054 13537 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:31.198959 13874 consensus_queue.cc:260] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.199985 14330 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:31.200330 14330 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:31.200673 14330 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.201906 13874 ts_tablet_manager.cc:1428] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260807 08:00:31.203069 13874 tablet_bootstrap.cc:492] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:31.205209 13884 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.205693 13884 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.207510 13884 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.209062 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:31.210376 13537 leader_election.cc:341] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.211778 13876 tablet_bootstrap.cc:492] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:31.212260 13876 ts_tablet_manager.cc:1397] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.019s
I20260807 08:00:31.212848 14330 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.214978 13876 raft_consensus.cc:348] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.216048 13876 raft_consensus.cc:374] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.216305 13876 raft_consensus.cc:729] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.217015 13876 consensus_queue.cc:260] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.218014 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:31.218632 13807 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.219007 14330 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.219751 13874 tablet_bootstrap.cc:654] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.223347 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.223450 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.227741 13807 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:31.228958 13537 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:31.229794 14330 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:31.230624 14330 raft_consensus.cc:686] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:31.231340 13876 ts_tablet_manager.cc:1428] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260807 08:00:31.231419 14330 consensus_queue.cc:237] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.232316 13876 tablet_bootstrap.cc:492] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:31.239213 13876 tablet_bootstrap.cc:654] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.242058 13591 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.258164 14330 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.258670 14330 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:31.263590 13540 leader_election.cc:341] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.264250 13540 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:31.265184 13884 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 08:00:31.266530 13540 leader_election.cc:341] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.276538 14330 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.279093 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:31.288316 14049 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.309043 14049 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.310001 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:31.311210 14049 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:00:31.312327 13537 leader_election.cc:341] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.302386 13884 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.312985 13884 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.316454 13884 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.317961 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:31.326015 14105 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.326490 14105 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.329392 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:31.334656 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.337922 14105 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.339143 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:31.340883 13657 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:31.342109 14105 raft_consensus.cc:2726] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:31.360596 14105 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.361053 14105 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.362919 14105 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.364056 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:31.365799 13657 leader_election.cc:341] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.369557 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.370584 13706 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:31.371851 14049 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.372560 14049 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.375523 14049 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:31.379443 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:31.380401 13590 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:31.414402 13661 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:31.415433 14105 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260807 08:00:31.385769 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 45039.1us
I20260807 08:00:31.396083 13884 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.415903 14105 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:31.416410 14105 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.416414 13884 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.418465 13884 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:00:31.419406 13769 leader_election.cc:341] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.421720 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:31.426064 13537 leader_election.cc:341] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.425843 14105 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.431754 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:31.432901 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:31.433167 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:31.434132 13591 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:31.436065 13657 leader_election.cc:341] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.441536 14105 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.456521 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.456488 13874 tablet_bootstrap.cc:492] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:31.458252 13874 ts_tablet_manager.cc:1397] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.255s user 0.033s sys 0.072s
I20260807 08:00:31.461761 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.461848 13874 raft_consensus.cc:348] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.462770 13874 raft_consensus.cc:374] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.463070 13874 raft_consensus.cc:729] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.463685 13874 consensus_queue.cc:260] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.465960 13874 ts_tablet_manager.cc:1428] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 08:00:31.466857 13874 tablet_bootstrap.cc:492] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
W20260807 08:00:31.478674 13540 leader_election.cc:341] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.481490 14351 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.484899 14351 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.486861 14351 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.499723 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.503439 13540 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:31.508955 13591 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:31.509661 13874 tablet_bootstrap.cc:654] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.513489 13540 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:31.515116 13884 raft_consensus.cc:2726] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:00:31.511747 14339 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.519003 13661 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:31.515084 14330 raft_consensus.cc:2741] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260807 08:00:31.522284 13430 catalog_manager.cc:5495] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:31.527446 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:31.521924 14339 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.528739 13875 tablet_bootstrap.cc:492] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:31.529173 13875 ts_tablet_manager.cc:1397] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.383s user 0.099s sys 0.172s
I20260807 08:00:31.529659 14339 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:31.529858 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.530522 13706 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
W20260807 08:00:31.531635 13540 leader_election.cc:341] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.532680 13540 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:31.521888 14346 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:31.533573 13884 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:31.534920 13540 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:31.537845 13963 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:31.539726 13963 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:31.540059 13963 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.543779 14253 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:31.549060 13875 raft_consensus.cc:348] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.549991 13875 raft_consensus.cc:374] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.537488 14346 raft_consensus.cc:686] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:31.550323 13875 raft_consensus.cc:729] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.551689 13875 consensus_queue.cc:260] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.555827 13875 ts_tablet_manager.cc:1428] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260807 08:00:31.556710 13875 tablet_bootstrap.cc:492] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:31.556522 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:31.557169 13807 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:31.561061 13963 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.555970 14346 consensus_queue.cc:237] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:31.569864 13769 leader_election.cc:341] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.570297 13769 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:31.570983 14049 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:31.571291 14105 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:00:31.571367 13769 leader_election.cc:341] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.571851 14105 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.572389 14049 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:31.576587 13876 tablet_bootstrap.cc:492] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:31.577162 13876 ts_tablet_manager.cc:1397] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.345s user 0.064s sys 0.102s
I20260807 08:00:31.582723 13876 raft_consensus.cc:348] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.582901 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:31.583573 13876 raft_consensus.cc:374] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.583880 13876 raft_consensus.cc:729] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.584172 13592 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 1: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:00:31.584491 13876 consensus_queue.cc:260] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:31.586467 13769 leader_election.cc:341] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.585680 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.585840 13963 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.587139 13769 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:31.587669 14105 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.589313 13706 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:31.590440 14253 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:31.595988 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:31.596611 13807 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:31.601241 13876 ts_tablet_manager.cc:1428] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
I20260807 08:00:31.601373 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:31.602165 13876 tablet_bootstrap.cc:492] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:31.603837 13657 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:31.604617 13769 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:31.606220 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:31.606827 13705 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.610541 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:00:31.611920 13661 leader_election.cc:341] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.611140 13816 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.615576 14105 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:31.615953 14105 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:31.616319 14105 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.623970 14105 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.628258 14105 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:31.629191 13431 catalog_manager.cc:5495] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:31.632469 13874 tablet_bootstrap.cc:492] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:31.632928 13874 ts_tablet_manager.cc:1397] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.166s user 0.006s sys 0.014s
I20260807 08:00:31.637326 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:31.638185 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:31.638841 13811 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.638932 14253 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260807 08:00:31.643976 13874 raft_consensus.cc:348] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.644649 13874 raft_consensus.cc:374] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.644904 13874 raft_consensus.cc:729] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.645541 13816 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:31.655433 14049 raft_consensus.cc:2741] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260807 08:00:31.645491 13874 consensus_queue.cc:260] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.658023 13537 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:31.658475 13874 ts_tablet_manager.cc:1428] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.025s user 0.003s sys 0.002s
I20260807 08:00:31.658876 14330 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:31.659292 13874 tablet_bootstrap.cc:492] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:31.645448 13711 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:31.647487 13661 leader_election.cc:341] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.647989 13705 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:31.659435 14330 raft_consensus.cc:686] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:31.658943 13811 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:31.671355 13657 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:31.673298 14349 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:31.673813 14349 raft_consensus.cc:686] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:31.680454 14349 consensus_queue.cc:237] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.682991 13876 tablet_bootstrap.cc:654] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.683174 13874 tablet_bootstrap.cc:654] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.685803 14330 consensus_queue.cc:237] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.744700 13875 tablet_bootstrap.cc:654] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.773018 14411 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.773734 14411 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.780591 13874 tablet_bootstrap.cc:492] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:31.781325 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:31.787611 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.787819 13589 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:31.789029 13874 ts_tablet_manager.cc:1397] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.130s user 0.008s sys 0.004s
W20260807 08:00:31.789139 13769 leader_election.cc:341] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.790198 14411 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:31.791038 13769 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:31.793191 14409 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:31.793668 14409 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:31.794155 14409 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.783947 13431 catalog_manager.cc:5495] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:31.786090 13876 tablet_bootstrap.cc:492] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
W20260807 08:00:31.803045 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 90735.4us
I20260807 08:00:31.803321 13876 ts_tablet_manager.cc:1397] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.201s user 0.014s sys 0.001s
I20260807 08:00:31.804394 13875 tablet_bootstrap.cc:492] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:31.804795 13875 ts_tablet_manager.cc:1397] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.248s user 0.021s sys 0.002s
I20260807 08:00:31.813675 13875 raft_consensus.cc:348] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.814352 13875 raft_consensus.cc:374] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.814769 13875 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.815354 13875 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.817490 13711 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.819939 14351 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:31.826148 13875 ts_tablet_manager.cc:1428] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.021s user 0.003s sys 0.003s
I20260807 08:00:31.809306 13876 raft_consensus.cc:348] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.826514 13876 raft_consensus.cc:374] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.826898 13876 raft_consensus.cc:729] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.827045 13875 tablet_bootstrap.cc:492] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:31.827531 13876 consensus_queue.cc:260] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.804162 13874 raft_consensus.cc:348] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.829792 13874 raft_consensus.cc:374] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:31.830127 13874 raft_consensus.cc:729] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:31.830771 14330 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:31.801725 14409 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.837795 14049 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.840937 13876 ts_tablet_manager.cc:1428] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.037s user 0.004s sys 0.003s
I20260807 08:00:31.856751 13876 tablet_bootstrap.cc:492] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:31.844811 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:31.865322 13875 tablet_bootstrap.cc:654] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.842778 14049 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.868762 14049 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:31.868508 13428 catalog_manager.cc:5495] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:31.839967 14409 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:31.838032 13874 consensus_queue.cc:260] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:31.871802 13769 leader_election.cc:341] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.868508 13876 tablet_bootstrap.cc:654] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:31.840788 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:31.874425 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:31.904217 13592 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:31.876437 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.906126 13769 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:00:31.910593 13769 leader_election.cc:341] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:31.912865 14411 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:31.915486 14411 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:31.915901 14411 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:31.911309 13587 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:31.946578 13508 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 08:00:31.969835 14409 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:31.970455 14409 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.972412 14409 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:31.975937 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:31.976910 13592 raft_consensus.cc:2379] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 1: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:00:31.979231 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:31.986570 14411 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:31.989092 14411 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:31.993655 13587 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:32.012800 13876 tablet_bootstrap.cc:492] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:32.013423 13876 ts_tablet_manager.cc:1397] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.157s user 0.013s sys 0.036s
I20260807 08:00:32.000828 13769 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.015599 14409 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.016422 13876 raft_consensus.cc:348] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.017504 13876 raft_consensus.cc:374] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.017931 13876 raft_consensus.cc:729] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.020150 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:32.021685 13769 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.022893 14255 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:32.023339 14255 raft_consensus.cc:686] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:32.023389 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:32.024371 13587 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.024143 14255 consensus_queue.cc:237] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.007710 13874 ts_tablet_manager.cc:1428] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.218s user 0.007s sys 0.000s
I20260807 08:00:32.034511 13874 tablet_bootstrap.cc:492] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:32.054032 14409 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:32.115521 13884 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.117403 13884 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.123128 13874 tablet_bootstrap.cc:654] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.124084 13876 consensus_queue.cc:260] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:32.127185 13769 leader_election.cc:341] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.128680 13963 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.129176 13875 tablet_bootstrap.cc:492] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:32.132129 13875 ts_tablet_manager.cc:1397] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.305s user 0.049s sys 0.090s
I20260807 08:00:32.131932 13963 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.134752 13963 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.136662 13875 raft_consensus.cc:348] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.143919 13875 raft_consensus.cc:374] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.144280 13875 raft_consensus.cc:729] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.145161 13875 consensus_queue.cc:260] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.147047 13587 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:32.147478 13875 ts_tablet_manager.cc:1428] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.015s user 0.006s sys 0.002s
I20260807 08:00:32.149264 13875 tablet_bootstrap.cc:492] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:32.123009 13884 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.198518 13874 tablet_bootstrap.cc:492] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:32.199231 13874 ts_tablet_manager.cc:1397] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.165s user 0.019s sys 0.003s
I20260807 08:00:32.213748 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:32.215118 13816 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:32.217484 14454 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.218194 14454 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.218685 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:32.220659 14454 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:00:32.221290 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 46141.6us
I20260807 08:00:32.224920 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:32.227950 13874 raft_consensus.cc:348] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.228735 13874 raft_consensus.cc:374] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.229033 13874 raft_consensus.cc:729] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.229084 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:32.229774 13874 consensus_queue.cc:260] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.230542 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:32.230798 13816 raft_consensus.cc:2454] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:32.232724 13876 ts_tablet_manager.cc:1428] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.219s user 0.009s sys 0.001s
I20260807 08:00:32.234318 13876 tablet_bootstrap.cc:492] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:32.238663 13537 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:32.240121 14424 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:32.240036 13769 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.243814 14453 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:32.245232 14453 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:32.246621 14453 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.249284 13874 ts_tablet_manager.cc:1428] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.050s user 0.000s sys 0.007s
I20260807 08:00:32.249552 13431 catalog_manager.cc:5495] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
W20260807 08:00:32.255051 13540 leader_election.cc:341] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.255986 14424 consensus_queue.cc:1035] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:32.258432 14454 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:32.259418 13874 tablet_bootstrap.cc:492] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
W20260807 08:00:32.260159 13540 leader_election.cc:341] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.264599 14049 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:32.265102 14049 raft_consensus.cc:686] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:32.265416 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:32.266137 14049 consensus_queue.cc:237] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:32.263731 13661 leader_election.cc:341] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.277912 13537 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.279146 14330 raft_consensus.cc:2781] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:32.279549 14330 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:32.279917 14330 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:32.304382 13657 leader_election.cc:341] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.304951 13657 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:32.314481 14445 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:32.316004 13876 tablet_bootstrap.cc:654] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.327699 14447 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.328303 14447 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:32.339216 13661 consensus_peers.cc:487] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Couldn't send request to peer 77da774f2bb54cf28302d53d0db224ad. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:32.343137 14447 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.347018 13875 tablet_bootstrap.cc:654] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.347074 13884 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:32.353231 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:32.357721 14453 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:32.361455 13661 leader_election.cc:341] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.362949 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:32.363808 13811 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.365865 14330 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.368582 13874 tablet_bootstrap.cc:654] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.369853 13884 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.378974 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:32.379119 13884 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.379585 13807 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.380928 13884 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.373831 14330 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.403015 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:32.404270 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:32.376484 14453 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.399626 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:32.407200 13540 leader_election.cc:341] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:32.411460 13540 leader_election.cc:341] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:32.414108 14464 log.cc:924] Time spent T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260807 08:00:32.418375 13807 raft_consensus.cc:2454] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:32.403049 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:00:32.424146 13537 leader_election.cc:341] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.424892 13537 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.426275 14351 raft_consensus.cc:2781] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:32.426759 14351 raft_consensus.cc:686] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:32.427660 14351 consensus_queue.cc:237] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.411826 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:32.437333 13884 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:32.476948 13875 tablet_bootstrap.cc:492] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:32.486676 13874 tablet_bootstrap.cc:492] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:32.486749 13875 ts_tablet_manager.cc:1397] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.338s user 0.020s sys 0.008s
I20260807 08:00:32.487203 13874 ts_tablet_manager.cc:1397] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.228s user 0.014s sys 0.007s
I20260807 08:00:32.489015 13875 raft_consensus.cc:348] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.489599 13875 raft_consensus.cc:374] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.489873 13875 raft_consensus.cc:729] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.490509 13875 consensus_queue.cc:260] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.492197 13875 ts_tablet_manager.cc:1428] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 08:00:32.493177 13875 tablet_bootstrap.cc:492] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:32.432823 13811 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
W20260807 08:00:32.468524 14463 log.cc:924] Time spent T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
I20260807 08:00:32.497481 13537 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.497298 13874 raft_consensus.cc:348] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.498263 13874 raft_consensus.cc:374] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.498587 13874 raft_consensus.cc:729] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.499897 13875 tablet_bootstrap.cc:654] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.499454 13874 consensus_queue.cc:260] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:32.481389 13657 leader_election.cc:341] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.502672 13657 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:00:32.482230 13540 leader_election.cc:341] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.482349 13876 tablet_bootstrap.cc:492] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:32.504443 13874 ts_tablet_manager.cc:1428] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260807 08:00:32.505071 13876 ts_tablet_manager.cc:1397] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.271s user 0.016s sys 0.007s
I20260807 08:00:32.506223 13874 tablet_bootstrap.cc:492] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:32.506999 13540 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.507707 14454 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:32.515544 13874 tablet_bootstrap.cc:654] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.517220 14450 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:32.518536 13884 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:32.518827 13876 raft_consensus.cc:348] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.520078 13876 raft_consensus.cc:374] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.520320 13876 raft_consensus.cc:729] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.521855 14450 raft_consensus.cc:686] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:32.522909 14450 consensus_queue.cc:237] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.520932 13876 consensus_queue.cc:260] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.526540 13876 ts_tablet_manager.cc:1428] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.021s user 0.004s sys 0.002s
I20260807 08:00:32.527098 14409 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:32.528236 13876 tablet_bootstrap.cc:492] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:32.532241 13874 tablet_bootstrap.cc:492] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:32.532702 13874 ts_tablet_manager.cc:1397] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.000s
I20260807 08:00:32.544569 13874 raft_consensus.cc:348] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.552204 13874 raft_consensus.cc:374] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.552511 13874 raft_consensus.cc:729] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.553114 13874 consensus_queue.cc:260] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.561851 13875 tablet_bootstrap.cc:492] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:32.562292 13875 ts_tablet_manager.cc:1397] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.069s user 0.040s sys 0.010s
I20260807 08:00:32.562876 13876 tablet_bootstrap.cc:654] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.552873 13431 catalog_manager.cc:5495] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:32.566984 13874 ts_tablet_manager.cc:1428] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.034s user 0.008s sys 0.000s
I20260807 08:00:32.567973 13874 tablet_bootstrap.cc:492] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:32.593106 13428 catalog_manager.cc:5495] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:32.598716 13875 raft_consensus.cc:348] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.599439 13875 raft_consensus.cc:374] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.599702 13875 raft_consensus.cc:729] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.605940 14049 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.602299 13874 tablet_bootstrap.cc:654] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
W20260807 08:00:32.606285 13769 consensus_peers.cc:487] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Couldn't send request to peer 9210dbc4cc9c45e5b26a463b705d362c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:32.605021 14105 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.606513 14049 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.607172 14105 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.608961 14105 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.608989 14049 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:32.606535 13875 consensus_queue.cc:260] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.615219 13875 ts_tablet_manager.cc:1428] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.052s user 0.003s sys 0.002s
I20260807 08:00:32.605859 13876 tablet_bootstrap.cc:492] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:32.615724 13876 ts_tablet_manager.cc:1397] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.088s user 0.013s sys 0.012s
I20260807 08:00:32.611642 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:32.612627 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:32.617457 13711 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:32.613016 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:32.610889 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:32.618523 13769 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:32.618747 13589 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:32.619920 13875 tablet_bootstrap.cc:492] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:32.620091 14255 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:32.620501 14255 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:32.620815 14255 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.622138 13661 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:00:32.622642 13769 leader_election.cc:341] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.622915 14105 raft_consensus.cc:2781] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:32.623235 14105 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:32.623533 14105 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:32.625422 13657 leader_election.cc:341] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.618858 13876 raft_consensus.cc:348] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.626255 13876 raft_consensus.cc:374] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.626531 13876 raft_consensus.cc:729] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.627233 13876 consensus_queue.cc:260] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.629309 13876 ts_tablet_manager.cc:1428] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260807 08:00:32.625572 13875 tablet_bootstrap.cc:654] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.639994 13874 tablet_bootstrap.cc:492] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:32.633467 14255 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.641655 13874 ts_tablet_manager.cc:1397] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.074s user 0.015s sys 0.000s
I20260807 08:00:32.643558 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:32.643658 13874 raft_consensus.cc:348] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.644273 13874 raft_consensus.cc:374] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.644541 13874 raft_consensus.cc:729] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
W20260807 08:00:32.644816 13769 leader_election.cc:341] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.641052 14105 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.645329 13874 consensus_queue.cc:260] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.647507 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:32.647878 14105 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.648056 13587 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.635403 13876 tablet_bootstrap.cc:492] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:32.649441 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:32.650156 13711 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.648628 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:00:32.654410 13657 leader_election.cc:341] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.656173 13587 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:32.657172 13661 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:32.658926 14454 raft_consensus.cc:2781] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:32.659149 13711 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:32.660292 13769 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:32.660957 13876 tablet_bootstrap.cc:654] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.661010 14049 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:32.661599 14049 raft_consensus.cc:686] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:32.661887 14454 raft_consensus.cc:686] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:32.662420 14049 consensus_queue.cc:237] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.662701 14454 consensus_queue.cc:237] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.662473 13431 catalog_manager.cc:5495] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:32.649482 14255 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:32.666929 13874 ts_tablet_manager.cc:1428] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260807 08:00:32.668002 13874 tablet_bootstrap.cc:492] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:32.668583 13876 tablet_bootstrap.cc:492] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:32.668982 13876 ts_tablet_manager.cc:1397] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.008s
I20260807 08:00:32.634712 13875 tablet_bootstrap.cc:492] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:32.670856 13875 ts_tablet_manager.cc:1397] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.004s
I20260807 08:00:32.671571 13876 raft_consensus.cc:348] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.672878 13876 raft_consensus.cc:374] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.674156 13875 raft_consensus.cc:348] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.674994 13875 raft_consensus.cc:374] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.675433 13875 raft_consensus.cc:729] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.676334 13875 consensus_queue.cc:260] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.676975 13874 tablet_bootstrap.cc:654] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.678743 13875 ts_tablet_manager.cc:1428] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260807 08:00:32.679768 13875 tablet_bootstrap.cc:492] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:32.673131 13876 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.681126 13876 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.684132 13876 ts_tablet_manager.cc:1428] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260807 08:00:32.685039 13876 tablet_bootstrap.cc:492] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:32.687855 13875 tablet_bootstrap.cc:654] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.692256 14049 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.692842 14049 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.695604 14049 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:32.695865 13874 tablet_bootstrap.cc:492] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:32.697052 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:32.698216 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:32.710112 13587 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
W20260807 08:00:32.710325 13769 leader_election.cc:341] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.706231 13876 tablet_bootstrap.cc:654] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.709244 13874 ts_tablet_manager.cc:1397] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.002s
I20260807 08:00:32.713255 13874 raft_consensus.cc:348] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.713989 13874 raft_consensus.cc:374] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.714250 13874 raft_consensus.cc:729] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.714905 13874 consensus_queue.cc:260] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.715368 13769 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.714833 13428 catalog_manager.cc:5495] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:32.717031 14255 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:32.717127 13874 ts_tablet_manager.cc:1428] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 08:00:32.717410 14255 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:32.717895 14255 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.721262 13875 tablet_bootstrap.cc:492] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:32.721486 13874 tablet_bootstrap.cc:492] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:32.721719 13875 ts_tablet_manager.cc:1397] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.006s
I20260807 08:00:32.724231 13875 raft_consensus.cc:348] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.724972 13875 raft_consensus.cc:374] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.725242 13875 raft_consensus.cc:729] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.726046 13875 consensus_queue.cc:260] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:32.729197 13769 consensus_peers.cc:487] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Couldn't send request to peer 9210dbc4cc9c45e5b26a463b705d362c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:32.728701 13430 catalog_manager.cc:5495] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:32.730883 13874 tablet_bootstrap.cc:654] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.732366 14255 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.734328 14255 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:32.734978 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:32.735586 13587 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.736541 13875 ts_tablet_manager.cc:1428] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.015s user 0.002s sys 0.004s
I20260807 08:00:32.737501 13875 tablet_bootstrap.cc:492] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:32.737862 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:00:32.741775 13769 leader_election.cc:341] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.747447 13875 tablet_bootstrap.cc:654] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.747540 13587 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:32.749661 13769 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.750429 14409 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:32.751041 14409 raft_consensus.cc:686] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:32.754314 13876 tablet_bootstrap.cc:492] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:32.759491 14049 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:32.759980 13876 ts_tablet_manager.cc:1397] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.075s user 0.011s sys 0.002s
I20260807 08:00:32.754544 14409 consensus_queue.cc:237] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.763024 13876 raft_consensus.cc:348] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.763720 13876 raft_consensus.cc:374] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.764046 13876 raft_consensus.cc:729] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.764868 13876 consensus_queue.cc:260] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.769899 13876 ts_tablet_manager.cc:1428] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20260807 08:00:32.770897 13876 tablet_bootstrap.cc:492] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:32.773983 13875 tablet_bootstrap.cc:492] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:32.774431 13875 ts_tablet_manager.cc:1397] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.011s
I20260807 08:00:32.776194 13876 tablet_bootstrap.cc:654] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.776836 13875 raft_consensus.cc:348] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.777638 13875 raft_consensus.cc:374] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.777904 13875 raft_consensus.cc:729] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.778546 13874 tablet_bootstrap.cc:492] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:32.778542 13875 consensus_queue.cc:260] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.779151 13874 ts_tablet_manager.cc:1397] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.011s
I20260807 08:00:32.779616 14330 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.781046 14330 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.784603 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:32.787221 13874 raft_consensus.cc:348] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.787760 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:32.787900 13874 raft_consensus.cc:374] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260807 08:00:32.788224 13537 leader_election.cc:341] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.788344 13711 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 0.
I20260807 08:00:32.788264 13874 raft_consensus.cc:729] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.789628 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:32.792040 13875 ts_tablet_manager.cc:1428] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260807 08:00:32.793576 14351 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:32.793936 14351 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:32.794231 14351 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.801620 13875 tablet_bootstrap.cc:492] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:32.806003 14105 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.806526 14105 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.820129 13875 tablet_bootstrap.cc:654] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:32.789125 14330 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.831857 13431 catalog_manager.cc:5495] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:32.837996 14454 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.838657 14454 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.840982 14454 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.842149 14105 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.843874 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:32.844568 13589 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:32.845113 13875 tablet_bootstrap.cc:492] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:32.845155 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:32.845711 13875 ts_tablet_manager.cc:1397] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.002s
I20260807 08:00:32.845754 13661 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:32.846688 14105 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:32.847050 14105 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:32.847493 14105 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.845858 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:32.849442 13875 raft_consensus.cc:348] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.850847 13811 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 0.
I20260807 08:00:32.851187 13875 raft_consensus.cc:374] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:32.852312 13875 raft_consensus.cc:729] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:32.853356 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:32.853957 13875 consensus_queue.cc:260] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.855032 13661 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:32.855437 13875 ts_tablet_manager.cc:1428] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260807 08:00:32.855746 14454 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:32.847083 13874 consensus_queue.cc:260] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.857872 14453 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:32.858466 13874 ts_tablet_manager.cc:1428] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.079s user 0.008s sys 0.000s
I20260807 08:00:32.859500 13874 tablet_bootstrap.cc:492] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: Bootstrap starting.
I20260807 08:00:32.863560 13875 tablet_bootstrap.cc:492] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:32.864596 14453 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.865111 14453 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.869220 14255 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.869750 14255 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.871377 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:32.871817 14255 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:32.872211 13875 tablet_bootstrap.cc:654] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
W20260807 08:00:32.873586 13540 leader_election.cc:341] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.873972 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:32.874451 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:32.875646 14447 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:00:32.876374 13769 leader_election.cc:341] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:32.876122 14447 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.877381 13587 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:32.878255 14447 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.878511 13769 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:32.879606 14255 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:32.879990 14255 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:32.880383 14255 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.881135 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:32.883108 14351 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.892620 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:32.892720 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:32.893533 13592 raft_consensus.cc:2379] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 1: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:00:32.894364 13708 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.902360 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:32.897732 14453 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.905054 13875 tablet_bootstrap.cc:492] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:32.905776 13661 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:32.906375 13875 ts_tablet_manager.cc:1397] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.006s
I20260807 08:00:32.906797 14447 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:32.925364 14349 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.926012 14349 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.934580 14105 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:32.964357 14105 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:32.973775 13708 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:00:32.992379 14105 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:32.992918 14105 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.036286 14351 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:33.119478 13874 tablet_bootstrap.cc:654] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:33.126082 13876 tablet_bootstrap.cc:492] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:33.126561 13876 ts_tablet_manager.cc:1397] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.356s user 0.039s sys 0.044s
I20260807 08:00:33.123340 14105 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:33.128901 13876 raft_consensus.cc:348] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.129688 13876 raft_consensus.cc:374] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:33.123849 14447 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:33.129966 13876 raft_consensus.cc:729] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:33.130589 13876 consensus_queue.cc:260] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.131239 14411 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:33.132005 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:33.131745 14411 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.133826 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:33.134327 14411 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:33.134516 13811 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.138571 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:33.138885 13876 ts_tablet_manager.cc:1428] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.012s user 0.000s sys 0.009s
I20260807 08:00:33.139150 13592 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.139955 13876 tablet_bootstrap.cc:492] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
I20260807 08:00:33.142465 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:33.143251 13814 raft_consensus.cc:2379] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 1: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:00:33.145658 13876 tablet_bootstrap.cc:654] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:33.145728 13592 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:33.152518 14255 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.154338 13817 raft_consensus.cc:2379] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 1: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:00:33.155682 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:33.157119 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:33.158195 13875 raft_consensus.cc:348] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.159652 14349 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:33.159835 13875 raft_consensus.cc:374] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:33.171255 13875 raft_consensus.cc:729] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:33.164224 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:33.171960 13875 consensus_queue.cc:260] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.176332 13963 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:33.176556 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:33.177029 13963 raft_consensus.cc:686] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:33.177248 13589 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 0.
I20260807 08:00:33.178313 13963 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.182085 14253 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:33.182828 14253 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.191169 14253 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:33.210497 14255 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:33.216056 13811 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:00:33.217061 13874 tablet_bootstrap.cc:492] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: No bootstrap required, opened a new log
I20260807 08:00:33.216899 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:33.226537 13874 ts_tablet_manager.cc:1397] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: Time spent bootstrapping tablet: real 0.367s user 0.004s sys 0.010s
I20260807 08:00:33.229115 13874 raft_consensus.cc:348] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.229898 13874 raft_consensus.cc:374] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:33.230261 13874 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Initialized, Role: FOLLOWER
I20260807 08:00:33.232755 13876 tablet_bootstrap.cc:492] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:33.233193 13876 ts_tablet_manager.cc:1397] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.093s user 0.016s sys 0.000s
W20260807 08:00:33.233429 13508 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260807 08:00:33.234340 13874 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.236148 13711 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.246551 13874 ts_tablet_manager.cc:1428] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: Time spent starting tablet: real 0.020s user 0.002s sys 0.004s
I20260807 08:00:33.237661 13876 raft_consensus.cc:348] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.246805 13661 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:33.274055 13876 raft_consensus.cc:374] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:33.268693 13431 catalog_manager.cc:5495] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:33.274492 13876 raft_consensus.cc:729] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:33.267304 14512 consensus_queue.cc:1035] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260807 08:00:33.275174 13876 consensus_queue.cc:260] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.271085 13875 ts_tablet_manager.cc:1428] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.364s user 0.006s sys 0.022s
I20260807 08:00:33.277284 13876 ts_tablet_manager.cc:1428] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.044s user 0.017s sys 0.014s
W20260807 08:00:33.278056 13540 consensus_peers.cc:487] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad -> Peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Couldn't send request to peer 9210dbc4cc9c45e5b26a463b705d362c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:33.278239 13876 tablet_bootstrap.cc:492] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad: Bootstrap starting.
W20260807 08:00:33.257262 13537 leader_election.cc:341] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:33.279580 13537 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:33.280418 14330 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.284169 13876 tablet_bootstrap.cc:654] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:33.287585 13875 tablet_bootstrap.cc:492] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:33.289726 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:33.291541 14330 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 08:00:33.267940 13769 leader_election.cc:341] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
I20260807 08:00:33.295758 14516 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:33.296617 13876 tablet_bootstrap.cc:492] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad: No bootstrap required, opened a new log
I20260807 08:00:33.296484 14516 raft_consensus.cc:686] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:33.297127 13876 ts_tablet_manager.cc:1397] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.008s
I20260807 08:00:33.297847 13875 tablet_bootstrap.cc:654] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:33.297452 14516 consensus_queue.cc:237] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.299010 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:33.299952 13589 raft_consensus.cc:2379] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 1: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:00:33.300233 13876 raft_consensus.cc:348] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:00:33.301084 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 54378.9us
I20260807 08:00:33.300930 13876 raft_consensus.cc:374] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:33.303709 13876 raft_consensus.cc:729] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Initialized, Role: FOLLOWER
I20260807 08:00:33.304311 13876 consensus_queue.cc:260] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.305317 13661 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:33.308113 13816 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.310940 13711 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.311452 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:33.308279 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:33.318177 13875 tablet_bootstrap.cc:492] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:33.318586 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:33.319486 13705 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.319057 13875 ts_tablet_manager.cc:1397] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.003s
I20260807 08:00:33.323302 13875 raft_consensus.cc:348] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.323992 13875 raft_consensus.cc:374] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:33.324293 13875 raft_consensus.cc:729] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:33.325613 13875 consensus_queue.cc:260] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.310243 13876 ts_tablet_manager.cc:1428] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260807 08:00:33.330669 13875 ts_tablet_manager.cc:1428] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260807 08:00:33.331681 13875 tablet_bootstrap.cc:492] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c: Bootstrap starting.
I20260807 08:00:33.331873 13585 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:33.337383 13963 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:33.337927 13963 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.339805 13589 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.339881 13963 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:33.343721 14445 consensus_queue.cc:1035] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:33.348215 13657 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:33.350256 14514 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:33.349666 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:33.351619 13705 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:33.353233 14445 consensus_queue.cc:1035] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:33.371387 14447 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:33.356762 13711 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.371871 14447 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.373209 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:33.374008 14447 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:33.357158 13875 tablet_bootstrap.cc:654] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c: Neither blocks nor log segments found. Creating new log.
I20260807 08:00:33.360733 14509 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:33.376506 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:33.370097 13769 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:33.379122 14519 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:00:33.379580 14519 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:00:33.380008 14519 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.405323 13769 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:33.408655 14339 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:33.409185 14339 raft_consensus.cc:686] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:33.410323 14339 consensus_queue.cc:237] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.416320 14518 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:33.416867 13431 catalog_manager.cc:5495] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:33.420639 14519 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.432590 14519 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:33.455020 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:33.469170 14512 consensus_queue.cc:1035] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260807 08:00:33.458918 13769 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:33.464593 13587 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.475556 13711 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.480038 13537 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:33.482568 14518 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:33.527101 14255 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:33.529187 13657 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:33.539515 14509 raft_consensus.cc:2726] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:33.544450 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:33.542609 14330 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:33.548609 13875 tablet_bootstrap.cc:492] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c: No bootstrap required, opened a new log
I20260807 08:00:33.549093 13875 ts_tablet_manager.cc:1397] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent bootstrapping tablet: real 0.218s user 0.019s sys 0.038s
I20260807 08:00:33.553927 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:33.554582 13591 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:00:33.609944 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 48772.4us
I20260807 08:00:33.595573 13591 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:00:33.666154 14519 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:33.666643 14519 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.668553 14519 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:33.669085 14519 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 08:00:33.677917 13769 leader_election.cc:341] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Illegal state: must be running to vote when last-logged opid is not known
W20260807 08:00:33.679916 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 124708us
I20260807 08:00:33.686327 13875 raft_consensus.cc:348] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.687018 13875 raft_consensus.cc:374] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:00:33.687285 13875 raft_consensus.cc:729] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Initialized, Role: FOLLOWER
I20260807 08:00:33.687913 13875 consensus_queue.cc:260] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.690284 13875 ts_tablet_manager.cc:1428] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent starting tablet: real 0.141s user 0.015s sys 0.000s
I20260807 08:00:33.695533 14519 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:33.704020 14330 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:33.710652 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:00:33.718879 13769 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:33.725764 14339 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:00:33.727240 14339 raft_consensus.cc:686] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:33.728708 14339 consensus_queue.cc:237] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:33.738783 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:33.733374 13430 catalog_manager.cc:5495] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:33.749459 13769 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:00:33.750216 14409 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:33.763677 13814 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:33.770509 14454 consensus_queue.cc:1035] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:33.777495 14330 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:33.793254 13428 catalog_manager.cc:5495] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:33.801679 14454 consensus_queue.cc:1035] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:33.875418 14519 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:33.884923 14409 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:34.085673 14445 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:00:34.086210 14445 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:34.088100 14445 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:34.088593 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:00:34.089306 13591 raft_consensus.cc:2379] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 1: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:00:34.089363 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:00:34.094214 13657 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:34.094902 14445 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:00:34.103027 14445 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 08:00:34.194020 14519 consensus_queue.cc:1035] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:00:34.197911 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:00:34.219818 14565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:00:34.227447 14253 consensus_queue.cc:1035] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 08:00:34.234493 14566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:00:34.238137 14568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:00:34.258503 11825 server_base.cc:1034] running on GCE node
I20260807 08:00:34.265379 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260807 08:00:34.265820 11825 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260807 08:00:34.266759 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786089634266737 us; error 0 us; skew 500 ppm
I20260807 08:00:34.267513 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 08:00:34.277333 11825 webserver.cc:460] Webserver started at http://127.11.140.68:38337/ using document root <none> and password file <none>
I20260807 08:00:34.278988 11825 fs_manager.cc:359] Metadata directory not provided
I20260807 08:00:34.279256 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:00:34.279698 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:00:34.281448 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-3-root/instance:
uuid: "22ff0a1f4a4241d794ab182432c85893"
format_stamp: "Formatted at 2026-08-07 08:00:34 on dist-test-slave-zr1t"
I20260807 08:00:34.287128 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 08:00:34.291206 14578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:34.292174 11825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 08:00:34.292507 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-3-root
uuid: "22ff0a1f4a4241d794ab182432c85893"
format_stamp: "Formatted at 2026-08-07 08:00:34 on dist-test-slave-zr1t"
I20260807 08:00:34.292852 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:00:34.345201 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:00:34.346663 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:00:34.348623 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:00:34.351536 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:00:34.351791 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:34.352109 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:00:34.352319 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:00:34.440521 11825 rpc_server.cc:307] RPC server started. Bound to: 127.11.140.68:40123
I20260807 08:00:34.440641 14669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.140.68:40123 every 8 connection(s)
I20260807 08:00:34.454619 11825 auto_leader_rebalancer-test.cc:413] add a tserver: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:34.463618 14670 heartbeater.cc:347] Connected to a master server at 127.11.140.126:41033
I20260807 08:00:34.464082 14670 heartbeater.cc:464] Registering TS with master...
I20260807 08:00:34.464952 14670 heartbeater.cc:511] Master 127.11.140.126:41033 requested a full tablet report, sending...
I20260807 08:00:34.467065 13430 ts_manager.cc:194] Registered new tserver with Master: 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:00:34.469556 13430 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57868
I20260807 08:00:35.393232 13592 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:35.399427 13816 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:35.402603 14453 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:35.409482 13703 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:35.414718 14453 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260807 08:00:35.418284 14684 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.425398 13816 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.432322 13703 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.443341 13430 catalog_manager.cc:5495] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260807 08:00:35.445223 13540 consensus_peers.cc:487] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4614fbd9fc524f61bb9d11ea324e783e. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:35.472772 14670 heartbeater.cc:503] Master 127.11.140.126:41033 was elected leader, sending a full tablet report...
I20260807 08:00:35.468743 13816 consensus_queue.cc:237] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:35.524000 13711 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:35.526610 14253 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260807 08:00:35.678164 13585 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:35.721958 14518 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 08:00:35.755525 13769 consensus_peers.cc:487] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00c1526fe2ae48f3a1bcce761ff91f72. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:35.762291 14708 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.765295 13711 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.777631 13807 consensus_queue.cc:237] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:35.799424 13431 catalog_manager.cc:5495] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260807 08:00:35.804890 13769 consensus_peers.cc:487] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d324e0b3eeee43549f0b430264c5d25b. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:35.806355 13711 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:35.807608 14694 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:35.819298 13591 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:35.823079 13591 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.826745 14708 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:35.828253 14694 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.841404 13711 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.842379 13591 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.855875 13428 catalog_manager.cc:5495] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:35.890691 13703 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:35.897790 13807 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:35.899369 14509 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:35.901486 13591 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:35.902762 14509 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:35.908962 14696 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.910673 13811 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
W20260807 08:00:35.922999 13661 consensus_peers.cc:487] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e72e5415e504959b4763e580400f8ee. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:35.933476 13430 catalog_manager.cc:5495] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:35.936650 13592 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:36.017485 14734 ts_tablet_manager.cc:927] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:36.019888 14735 ts_tablet_manager.cc:927] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:36.020628 14734 tablet_copy_client.cc:316] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:36.021890 14735 tablet_copy_client.cc:316] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:36.055953 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet 00c1526fe2ae48f3a1bcce761ff91f72 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:36.056577 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet 00c1526fe2ae48f3a1bcce761ff91f72 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-00c1526fe2ae48f3a1bcce761ff91f72
I20260807 08:00:36.057183 13608 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet 4614fbd9fc524f61bb9d11ea324e783e from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:36.057863 13608 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet 4614fbd9fc524f61bb9d11ea324e783e from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-4614fbd9fc524f61bb9d11ea324e783e
I20260807 08:00:36.064131 13830 tablet_copy_source_session.cc:205] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:36.064210 13608 tablet_copy_source_session.cc:205] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:36.068229 14734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c1526fe2ae48f3a1bcce761ff91f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:36.068948 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4614fbd9fc524f61bb9d11ea324e783e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:36.082381 14734 tablet_copy_client.cc:791] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:36.082999 14734 tablet_copy_client.cc:655] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:36.082381 14735 tablet_copy_client.cc:791] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:36.083976 14735 tablet_copy_client.cc:655] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:36.087947 14735 tablet_copy_client.cc:531] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:36.088318 14734 tablet_copy_client.cc:531] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:36.095691 14735 tablet_bootstrap.cc:492] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:36.097666 14734 tablet_bootstrap.cc:492] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:36.123746 14735 tablet_bootstrap.cc:492] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:36.124734 14735 tablet_bootstrap.cc:492] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:36.125391 14735 ts_tablet_manager.cc:1397] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.009s
I20260807 08:00:36.128162 14735 raft_consensus.cc:348] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:36.129041 14735 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:36.129709 14735 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:36.136837 14735 ts_tablet_manager.cc:1428] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260807 08:00:36.138633 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-4614fbd9fc524f61bb9d11ea324e783e received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:36.139056 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-4614fbd9fc524f61bb9d11ea324e783e on tablet 4614fbd9fc524f61bb9d11ea324e783e with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:36.139851 14734 tablet_bootstrap.cc:492] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:36.140785 14734 tablet_bootstrap.cc:492] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:36.141827 14734 ts_tablet_manager.cc:1397] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.044s user 0.021s sys 0.014s
I20260807 08:00:36.144378 14734 raft_consensus.cc:348] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:36.145221 14734 raft_consensus.cc:729] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:36.145831 14734 consensus_queue.cc:260] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:36.147929 14734 ts_tablet_manager.cc:1428] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260807 08:00:36.149408 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-00c1526fe2ae48f3a1bcce761ff91f72 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:36.149814 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-00c1526fe2ae48f3a1bcce761ff91f72 on tablet 00c1526fe2ae48f3a1bcce761ff91f72 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:36.344623 14734 ts_tablet_manager.cc:927] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:36.346417 14734 tablet_copy_client.cc:316] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:36.347901 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet d324e0b3eeee43549f0b430264c5d25b from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:36.348332 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet d324e0b3eeee43549f0b430264c5d25b from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-d324e0b3eeee43549f0b430264c5d25b
I20260807 08:00:36.354864 13830 tablet_copy_source_session.cc:205] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:36.357641 14734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d324e0b3eeee43549f0b430264c5d25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:36.370941 14734 tablet_copy_client.cc:791] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:36.371255 14735 ts_tablet_manager.cc:927] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:36.371570 14734 tablet_copy_client.cc:655] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:36.375846 14734 tablet_copy_client.cc:531] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:36.377475 14735 tablet_copy_client.cc:316] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:00:36.388636 14734 tablet_bootstrap.cc:492] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:36.397178 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet 0e72e5415e504959b4763e580400f8ee from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:00:36.397842 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet 0e72e5415e504959b4763e580400f8ee from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-0e72e5415e504959b4763e580400f8ee
I20260807 08:00:36.404400 13724 tablet_copy_source_session.cc:205] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:36.407274 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e72e5415e504959b4763e580400f8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:36.410426 14734 tablet_bootstrap.cc:492] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:36.411268 14734 tablet_bootstrap.cc:492] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:36.411860 14734 ts_tablet_manager.cc:1397] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.000s
I20260807 08:00:36.415207 14734 raft_consensus.cc:348] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:36.416124 14734 raft_consensus.cc:729] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:36.416841 14734 consensus_queue.cc:260] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:36.419205 14734 ts_tablet_manager.cc:1428] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 08:00:36.426223 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-d324e0b3eeee43549f0b430264c5d25b received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:36.426641 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-d324e0b3eeee43549f0b430264c5d25b on tablet d324e0b3eeee43549f0b430264c5d25b with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:36.431571 14735 tablet_copy_client.cc:791] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:36.432116 14735 tablet_copy_client.cc:655] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:36.435448 14735 tablet_copy_client.cc:531] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:36.443879 14735 tablet_bootstrap.cc:492] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:36.461971 14735 tablet_bootstrap.cc:492] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:36.462855 14735 tablet_bootstrap.cc:492] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:36.463488 14735 ts_tablet_manager.cc:1397] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.001s
I20260807 08:00:36.465950 14735 raft_consensus.cc:348] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:36.466847 14735 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:36.467549 14735 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:36.469888 14735 ts_tablet_manager.cc:1428] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260807 08:00:36.472456 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-0e72e5415e504959b4763e580400f8ee received from {username='slave'} at 127.0.0.1:45382
I20260807 08:00:36.472807 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-0e72e5415e504959b4763e580400f8ee on tablet 0e72e5415e504959b4763e580400f8ee with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:36.514832 14630 raft_consensus.cc:1203] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:36.558488 14630 raft_consensus.cc:1203] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260807 08:00:36.635903 14664 debug-util.cc:398] Leaking SignalData structure 0x7b08002fefe0 after lost signal to thread 11828
W20260807 08:00:36.636775 14664 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff6c0 after lost signal to thread 13481
I20260807 08:00:36.875270 14630 raft_consensus.cc:1203] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:36.984715 14630 raft_consensus.cc:1203] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:37.025319 14764 raft_consensus.cc:1050] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:37.028347 14764 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:37.044147 14630 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.046389 13703 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.048277 14764 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.049587 13811 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.051709 14763 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:37.054318 14715 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.078402 14764 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.079885 13703 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.105190 13431 catalog_manager.cc:5495] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.095615 13809 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.127882 14630 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.147964 14767 raft_consensus.cc:1050] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:37.160770 14767 consensus_queue.cc:237] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:37.180675 14630 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.182598 14253 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.184705 13589 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.186448 14253 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.186046 13703 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.202615 14253 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.212702 14767 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.232931 13589 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.240394 13592 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:37.241506 14630 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.253427 13430 catalog_manager.cc:5495] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:37.254261 13711 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.258038 14629 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:37.258358 13703 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:37.259642 14763 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:37.267589 14763 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:37.291577 14772 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.297214 13705 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.299276 14630 raft_consensus.cc:2932] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.325376 13432 catalog_manager.cc:5495] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.328662 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4614fbd9fc524f61bb9d11ea324e783e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:37.346663 13807 consensus_queue.cc:237] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:37.353222 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet 4614fbd9fc524f61bb9d11ea324e783e with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:37.353245 14630 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:37.368827 14792 tablet_replica.cc:330] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:00:37.370312 13703 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:37.371871 14790 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:37.373243 14253 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:37.382357 14717 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.383424 14767 raft_consensus.cc:1050] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:37.383281 13703 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.386230 14767 consensus_queue.cc:237] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:37.393193 14792 raft_consensus.cc:2229] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:37.393994 14792 raft_consensus.cc:2258] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:37.396384 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 00c1526fe2ae48f3a1bcce761ff91f72 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:37.396626 14630 raft_consensus.cc:2932] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.405694 13431 catalog_manager.cc:5495] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:37.408810 14792 ts_tablet_manager.cc:1905] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:37.419600 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c1526fe2ae48f3a1bcce761ff91f72 with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:00:37.434865 14795 tablet_replica.cc:330] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:00:37.440335 14630 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.440863 13592 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.441988 13703 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.444383 14717 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.448134 14792 ts_tablet_manager.cc:1918] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:37.448606 14792 log.cc:1196] T 4614fbd9fc524f61bb9d11ea324e783e P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/4614fbd9fc524f61bb9d11ea324e783e
I20260807 08:00:37.450465 14790 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.452116 14795 raft_consensus.cc:2229] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:37.453069 14795 raft_consensus.cc:2258] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:37.453635 14253 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.456225 14795 ts_tablet_manager.cc:1905] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:37.458544 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet 4614fbd9fc524f61bb9d11ea324e783e (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:37.481647 14795 ts_tablet_manager.cc:1918] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:37.482098 14795 log.cc:1196] T 00c1526fe2ae48f3a1bcce761ff91f72 P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/00c1526fe2ae48f3a1bcce761ff91f72
I20260807 08:00:37.483671 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet 00c1526fe2ae48f3a1bcce761ff91f72 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:37.484391 14790 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.496205 13592 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.496764 14630 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.498777 13705 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.513407 13428 catalog_manager.cc:5495] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:37.545920 14799 raft_consensus.cc:1050] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:37.548313 14799 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:37.555883 13811 consensus_queue.cc:237] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:37.563006 13592 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:37.563165 14629 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.564361 14630 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:37.567087 14509 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.568138 13592 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.568195 14790 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:37.571292 14696 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.573985 14253 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:37.577751 13811 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:37.583833 14509 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:37.602623 14253 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.605350 14629 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.607306 13589 raft_consensus.cc:2932] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.622234 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d324e0b3eeee43549f0b430264c5d25b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:37.628150 13430 catalog_manager.cc:5495] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:37.687270 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet d324e0b3eeee43549f0b430264c5d25b with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:00:37.695544 14812 tablet_replica.cc:330] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:37.702338 14696 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.727963 13592 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.743876 14812 raft_consensus.cc:2229] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:37.745018 14812 raft_consensus.cc:2258] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:37.751675 14812 ts_tablet_manager.cc:1905] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:37.758448 14629 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.762389 13430 catalog_manager.cc:5495] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.800251 13809 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.813319 13711 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:37.825412 13589 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:37.830533 14509 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:37.840255 14812 ts_tablet_manager.cc:1918] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:37.840775 14812 log.cc:1196] T d324e0b3eeee43549f0b430264c5d25b P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/d324e0b3eeee43549f0b430264c5d25b
I20260807 08:00:37.849458 14800 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.852277 14630 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:37.864707 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet d324e0b3eeee43549f0b430264c5d25b (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:37.873003 14509 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:00:37.879160 13592 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
W20260807 08:00:37.882123 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 76510us
I20260807 08:00:37.884788 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0e72e5415e504959b4763e580400f8ee with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:37.884143 14629 raft_consensus.cc:2932] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:37.902673 13430 catalog_manager.cc:5495] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:37.924562 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e72e5415e504959b4763e580400f8ee with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:37.925634 14792 tablet_replica.cc:330] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:00:37.926608 14792 raft_consensus.cc:2229] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:37.927278 14792 raft_consensus.cc:2258] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:37.930579 14792 ts_tablet_manager.cc:1905] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:37.947754 14792 ts_tablet_manager.cc:1918] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:37.948175 14792 log.cc:1196] T 0e72e5415e504959b4763e580400f8ee P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/0e72e5415e504959b4763e580400f8ee
I20260807 08:00:37.949764 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet 0e72e5415e504959b4763e580400f8ee (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:38.984155 13703 consensus_queue.cc:237] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:38.990244 13809 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:38.990772 13661 consensus_peers.cc:487] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1c0fb87eb274da9bfb29c3b9300dc1c. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:38.991719 14800 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:38.992189 13585 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:38.993757 14509 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:39.005126 14804 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.011371 13811 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.013643 13585 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.036330 13811 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.042754 13430 catalog_manager.cc:5495] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:39.074226 13705 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:39.081195 13592 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:39.089941 14790 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:39.094657 14717 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 08:00:39.095477 13769 consensus_peers.cc:487] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5c39633297442a8a7c63b17b17cbe54. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:39.110656 14717 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.114643 13585 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.131244 13811 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.148422 13592 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:39.152221 14790 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 08:00:39.154422 13769 consensus_peers.cc:487] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13ee9b6f7fa547a3badcb0eba0210119. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:39.158977 13705 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:39.161334 13428 catalog_manager.cc:5495] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:39.162294 14790 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:39.174286 13711 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.197983 14831 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.204684 13705 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.213888 13592 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.214425 13811 consensus_queue.cc:237] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.236769 13431 catalog_manager.cc:5495] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:39.246160 13705 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:39.249087 13769 consensus_peers.cc:487] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c02eb1166e17491f82cfbd9c5cd030a2. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:39.252451 14831 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:39.256352 13592 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:39.262236 14849 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:39.264963 14790 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.274868 13592 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.281098 13711 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.292645 13428 catalog_manager.cc:5495] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:39.422374 14853 ts_tablet_manager.cc:927] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:39.425480 14853 tablet_copy_client.cc:316] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:00:39.426998 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet b1c0fb87eb274da9bfb29c3b9300dc1c from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:00:39.427388 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet b1c0fb87eb274da9bfb29c3b9300dc1c from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-b1c0fb87eb274da9bfb29c3b9300dc1c
I20260807 08:00:39.435777 13724 tablet_copy_source_session.cc:205] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:39.438791 14853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c0fb87eb274da9bfb29c3b9300dc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:39.452728 14853 tablet_copy_client.cc:791] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:39.453282 14853 tablet_copy_client.cc:655] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:39.457348 14853 tablet_copy_client.cc:531] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:39.465369 14853 tablet_bootstrap.cc:492] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:39.485752 14853 tablet_bootstrap.cc:492] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:39.486675 14853 tablet_bootstrap.cc:492] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:39.487411 14853 ts_tablet_manager.cc:1397] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.011s
I20260807 08:00:39.490677 14853 raft_consensus.cc:348] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.491618 14853 raft_consensus.cc:729] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:39.492311 14853 consensus_queue.cc:260] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.500494 14853 ts_tablet_manager.cc:1428] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260807 08:00:39.506541 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-b1c0fb87eb274da9bfb29c3b9300dc1c received from {username='slave'} at 127.0.0.1:45382
I20260807 08:00:39.506983 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-b1c0fb87eb274da9bfb29c3b9300dc1c on tablet b1c0fb87eb274da9bfb29c3b9300dc1c with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:39.573500 14853 ts_tablet_manager.cc:927] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:39.581029 14853 tablet_copy_client.cc:316] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:39.582643 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet d5c39633297442a8a7c63b17b17cbe54 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:39.583277 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet d5c39633297442a8a7c63b17b17cbe54 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-d5c39633297442a8a7c63b17b17cbe54
I20260807 08:00:39.596504 13830 tablet_copy_source_session.cc:205] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:39.601929 14853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c39633297442a8a7c63b17b17cbe54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:39.615406 14853 tablet_copy_client.cc:791] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:39.615973 14853 tablet_copy_client.cc:655] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:39.622912 14853 tablet_copy_client.cc:531] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:39.636857 14853 tablet_bootstrap.cc:492] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:39.677742 14862 ts_tablet_manager.cc:927] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:39.684504 14853 tablet_bootstrap.cc:492] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:39.685730 14853 tablet_bootstrap.cc:492] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:39.686510 14853 ts_tablet_manager.cc:1397] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.050s user 0.031s sys 0.005s
I20260807 08:00:39.689409 14853 raft_consensus.cc:348] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.692046 14853 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:39.692729 14853 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.699642 14853 ts_tablet_manager.cc:1428] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.013s user 0.002s sys 0.004s
I20260807 08:00:39.706311 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-d5c39633297442a8a7c63b17b17cbe54 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:39.706743 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-d5c39633297442a8a7c63b17b17cbe54 on tablet d5c39633297442a8a7c63b17b17cbe54 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:39.714440 14862 tablet_copy_client.cc:316] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:39.718948 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet 13ee9b6f7fa547a3badcb0eba0210119 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:39.719514 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet 13ee9b6f7fa547a3badcb0eba0210119 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-13ee9b6f7fa547a3badcb0eba0210119
I20260807 08:00:39.729765 13830 tablet_copy_source_session.cc:205] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:39.734484 14862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ee9b6f7fa547a3badcb0eba0210119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:39.749012 14862 tablet_copy_client.cc:791] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:39.749584 14862 tablet_copy_client.cc:655] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:39.752657 14862 tablet_copy_client.cc:531] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:39.764839 14862 tablet_bootstrap.cc:492] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:39.785486 14862 tablet_bootstrap.cc:492] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:39.786327 14862 tablet_bootstrap.cc:492] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:39.786990 14862 ts_tablet_manager.cc:1397] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.014s
I20260807 08:00:39.790295 14862 raft_consensus.cc:348] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.791148 14862 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:39.794042 14862 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.796908 14862 ts_tablet_manager.cc:1428] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20260807 08:00:39.804448 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-13ee9b6f7fa547a3badcb0eba0210119 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:39.804844 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-13ee9b6f7fa547a3badcb0eba0210119 on tablet 13ee9b6f7fa547a3badcb0eba0210119 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:39.828490 14629 raft_consensus.cc:1203] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:39.851259 14862 ts_tablet_manager.cc:927] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:39.853811 14862 tablet_copy_client.cc:316] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:39.855326 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet c02eb1166e17491f82cfbd9c5cd030a2 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:39.855890 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet c02eb1166e17491f82cfbd9c5cd030a2 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-c02eb1166e17491f82cfbd9c5cd030a2
I20260807 08:00:39.864408 13830 tablet_copy_source_session.cc:205] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:39.867449 14862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02eb1166e17491f82cfbd9c5cd030a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:39.888337 14862 tablet_copy_client.cc:791] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:39.888914 14862 tablet_copy_client.cc:655] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:39.897763 14862 tablet_copy_client.cc:531] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:39.908236 14862 tablet_bootstrap.cc:492] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:39.938169 14862 tablet_bootstrap.cc:492] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:39.939100 14862 tablet_bootstrap.cc:492] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:39.939800 14862 ts_tablet_manager.cc:1397] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.012s
I20260807 08:00:39.942595 14862 raft_consensus.cc:348] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.943495 14862 raft_consensus.cc:729] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:39.944223 14862 consensus_queue.cc:260] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:39.946955 14862 ts_tablet_manager.cc:1428] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260807 08:00:39.949143 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-c02eb1166e17491f82cfbd9c5cd030a2 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:39.949563 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-c02eb1166e17491f82cfbd9c5cd030a2 on tablet c02eb1166e17491f82cfbd9c5cd030a2 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:40.011732 14629 raft_consensus.cc:1203] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:40.259057 14629 raft_consensus.cc:1203] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:40.303597 14868 raft_consensus.cc:1050] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:40.306654 14868 consensus_queue.cc:237] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:40.325981 13809 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:40.327484 14867 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:40.331212 14629 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:40.331629 13585 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:40.332630 14804 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:40.333496 14868 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:40.345697 14804 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.347783 14629 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.371690 13428 catalog_manager.cc:5495] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.368757 13585 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.387989 13811 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.384256 14629 raft_consensus.cc:1203] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:40.459169 13705 consensus_queue.cc:237] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:40.472604 14629 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:40.474440 13811 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:40.475559 14881 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:40.477531 14867 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:40.484666 14804 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.486438 13807 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.486562 14629 raft_consensus.cc:2932] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.495275 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b1c0fb87eb274da9bfb29c3b9300dc1c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:40.505062 13430 catalog_manager.cc:5495] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.516786 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet b1c0fb87eb274da9bfb29c3b9300dc1c with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:00:40.523614 14884 tablet_replica.cc:330] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:00:40.532169 14790 raft_consensus.cc:1050] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:40.534505 14884 raft_consensus.cc:2229] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:40.536623 14884 raft_consensus.cc:2258] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:40.540022 14884 ts_tablet_manager.cc:1905] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:40.534437 14790 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:40.558275 14629 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:40.560024 14884 ts_tablet_manager.cc:1918] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:40.560552 14884 log.cc:1196] T b1c0fb87eb274da9bfb29c3b9300dc1c P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/b1c0fb87eb274da9bfb29c3b9300dc1c
I20260807 08:00:40.560745 14844 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:40.562201 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet b1c0fb87eb274da9bfb29c3b9300dc1c (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:40.564558 13705 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:40.566603 14844 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:40.578274 13592 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:40.580327 14790 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:40.605940 14849 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.607707 14629 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.624228 13711 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.634405 13431 catalog_manager.cc:5495] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.678197 13592 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.679601 13811 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:40.695310 14629 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:40.696902 14844 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:40.698652 13592 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:40.700403 14790 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:40.715606 14896 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.732208 13592 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.732661 14629 raft_consensus.cc:2932] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.735345 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d5c39633297442a8a7c63b17b17cbe54 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:40.746464 13430 catalog_manager.cc:5495] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:40.765093 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c39633297442a8a7c63b17b17cbe54 with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:00:40.783345 14899 tablet_replica.cc:330] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:40.791528 14899 raft_consensus.cc:2229] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:40.793870 14899 raft_consensus.cc:2258] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:40.798166 14844 raft_consensus.cc:1050] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:40.799823 14899 ts_tablet_manager.cc:1905] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:40.801391 14844 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:40.823768 13705 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:40.823915 14629 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:40.826912 13592 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:40.830194 14844 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:40.830794 14899 ts_tablet_manager.cc:1918] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:40.831265 14899 log.cc:1196] T d5c39633297442a8a7c63b17b17cbe54 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/d5c39633297442a8a7c63b17b17cbe54
I20260807 08:00:40.832696 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet d5c39633297442a8a7c63b17b17cbe54 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:40.832815 14844 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:40.834266 14896 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:40.874374 14849 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.895728 13711 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.885093 14629 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.907588 13430 catalog_manager.cc:5495] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:40.936721 13592 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:40.966974 13811 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:40.974155 14629 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:40.978024 14844 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:00:40.976974 13592 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:40.987306 14790 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:41.001609 14896 raft_consensus.cc:1050] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:41.001428 14894 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.004307 14896 consensus_queue.cc:237] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:41.008209 13592 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.017443 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 13ee9b6f7fa547a3badcb0eba0210119 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:41.020017 14629 raft_consensus.cc:2932] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.031942 13432 catalog_manager.cc:5495] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:41.047034 14630 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:41.049060 13703 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:41.062495 13585 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:41.068553 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ee9b6f7fa547a3badcb0eba0210119 with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:00:41.069501 14899 tablet_replica.cc:330] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:41.070530 14899 raft_consensus.cc:2229] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:41.070451 14894 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:41.071271 14899 raft_consensus.cc:2258] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:41.076089 14899 ts_tablet_manager.cc:1905] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:41.083386 14894 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:41.089350 14896 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:41.100342 14899 ts_tablet_manager.cc:1918] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:41.102576 14899 log.cc:1196] T 13ee9b6f7fa547a3badcb0eba0210119 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/13ee9b6f7fa547a3badcb0eba0210119
I20260807 08:00:41.104660 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet 13ee9b6f7fa547a3badcb0eba0210119 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:41.115455 14849 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.126097 13592 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.138754 13703 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.142382 14629 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.148540 13432 catalog_manager.cc:5495] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.187402 13811 consensus_queue.cc:237] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:41.196211 13705 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:41.198104 14629 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:41.199882 14849 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:41.204933 14849 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:00:41.215878 14790 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.219399 13705 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.223729 14630 raft_consensus.cc:2932] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:41.235386 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c02eb1166e17491f82cfbd9c5cd030a2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:41.244423 13432 catalog_manager.cc:5495] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:41.273450 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet c02eb1166e17491f82cfbd9c5cd030a2 with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:00:41.275616 14915 tablet_replica.cc:330] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:00:41.276732 14915 raft_consensus.cc:2229] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:41.277839 14915 raft_consensus.cc:2258] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:41.282043 14915 ts_tablet_manager.cc:1905] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:41.299021 14915 ts_tablet_manager.cc:1918] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:41.299544 14915 log.cc:1196] T c02eb1166e17491f82cfbd9c5cd030a2 P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/c02eb1166e17491f82cfbd9c5cd030a2
I20260807 08:00:41.300980 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet c02eb1166e17491f82cfbd9c5cd030a2 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:42.368162 13705 consensus_queue.cc:237] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:42.385152 13811 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:42.386263 13584 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:42.386279 13661 consensus_peers.cc:487] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46d5ddf3f7d84e0ebda02d99abff4f0e. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:42.387666 14868 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:42.389056 14876 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:42.404201 14804 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.406001 13811 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.415298 13585 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.433359 13584 consensus_queue.cc:237] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:42.443410 13432 catalog_manager.cc:5495] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.455209 13811 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:42.464968 14905 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 08:00:42.467522 13540 consensus_peers.cc:487] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35cf7298a676441c8d12e04b066ee036. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:42.469430 13711 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:42.477731 14763 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260807 08:00:42.506271 14905 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.510079 13711 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.524544 13811 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.525242 13705 consensus_queue.cc:237] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:42.531997 13809 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:42.534639 13584 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:42.535357 13661 consensus_peers.cc:487] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25fcdcbbf735436893ce5c13fd2eb1f1. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:42.535578 14804 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:42.539125 14804 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:42.536772 13431 catalog_manager.cc:5495] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:42.561663 14804 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.573879 13809 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.593312 13584 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.594697 13428 catalog_manager.cc:5495] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.595449 13585 consensus_queue.cc:237] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:42.604780 13811 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:42.605566 13540 consensus_peers.cc:487] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4e01d60f0064bb3958a65250e430990. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:42.607131 13705 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:42.637130 14933 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:42.639897 14934 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:42.649147 14933 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.650549 13809 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.675462 13711 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:42.676471 13432 catalog_manager.cc:5495] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:42.921490 14948 ts_tablet_manager.cc:927] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:42.923828 14948 tablet_copy_client.cc:316] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:00:42.925673 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:00:42.926128 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet 46d5ddf3f7d84e0ebda02d99abff4f0e from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-46d5ddf3f7d84e0ebda02d99abff4f0e
I20260807 08:00:42.931108 14950 ts_tablet_manager.cc:927] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:42.935470 13724 tablet_copy_source_session.cc:205] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:42.937393 14950 tablet_copy_client.cc:316] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:42.938911 14948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:42.941023 13608 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet 35cf7298a676441c8d12e04b066ee036 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:42.941612 13608 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet 35cf7298a676441c8d12e04b066ee036 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-35cf7298a676441c8d12e04b066ee036
I20260807 08:00:42.948845 13608 tablet_copy_source_session.cc:205] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:42.953182 14950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cf7298a676441c8d12e04b066ee036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:42.954880 14948 tablet_copy_client.cc:791] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:42.955634 14948 tablet_copy_client.cc:655] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:42.967248 14948 tablet_copy_client.cc:531] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:42.968128 14950 tablet_copy_client.cc:791] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:42.968726 14950 tablet_copy_client.cc:655] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:42.973308 14950 tablet_copy_client.cc:531] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:42.977190 14948 tablet_bootstrap.cc:492] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:42.985667 14950 tablet_bootstrap.cc:492] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:43.007107 14948 tablet_bootstrap.cc:492] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:43.008076 14948 tablet_bootstrap.cc:492] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:43.008827 14948 ts_tablet_manager.cc:1397] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.007s
I20260807 08:00:43.012795 14948 raft_consensus.cc:348] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:43.013847 14948 raft_consensus.cc:729] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:43.014853 14950 tablet_bootstrap.cc:492] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:43.014487 14948 consensus_queue.cc:260] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:43.017782 14950 tablet_bootstrap.cc:492] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:43.018106 14948 ts_tablet_manager.cc:1428] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260807 08:00:43.018550 14950 ts_tablet_manager.cc:1397] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.033s user 0.030s sys 0.002s
I20260807 08:00:43.020906 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-46d5ddf3f7d84e0ebda02d99abff4f0e received from {username='slave'} at 127.0.0.1:45382
I20260807 08:00:43.021355 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-46d5ddf3f7d84e0ebda02d99abff4f0e on tablet 46d5ddf3f7d84e0ebda02d99abff4f0e with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:43.022990 14950 raft_consensus.cc:348] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:43.023952 14950 raft_consensus.cc:729] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:43.026405 14950 consensus_queue.cc:260] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:43.028954 14950 ts_tablet_manager.cc:1428] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260807 08:00:43.030426 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-35cf7298a676441c8d12e04b066ee036 received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:43.030790 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-35cf7298a676441c8d12e04b066ee036 on tablet 35cf7298a676441c8d12e04b066ee036 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:43.033634 14950 ts_tablet_manager.cc:927] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:43.041649 14950 tablet_copy_client.cc:316] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:00:43.043573 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet 25fcdcbbf735436893ce5c13fd2eb1f1 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:00:43.044238 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet 25fcdcbbf735436893ce5c13fd2eb1f1 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-25fcdcbbf735436893ce5c13fd2eb1f1
I20260807 08:00:43.060830 13724 tablet_copy_source_session.cc:205] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:43.065089 14950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fcdcbbf735436893ce5c13fd2eb1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:43.078142 14950 tablet_copy_client.cc:791] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:43.078655 14950 tablet_copy_client.cc:655] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:43.082160 14950 tablet_copy_client.cc:531] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:43.088949 14950 tablet_bootstrap.cc:492] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:43.111771 14948 ts_tablet_manager.cc:927] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:43.113409 14948 tablet_copy_client.cc:316] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:43.114986 13608 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet e4e01d60f0064bb3958a65250e430990 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:43.115525 13608 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet e4e01d60f0064bb3958a65250e430990 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-e4e01d60f0064bb3958a65250e430990
I20260807 08:00:43.116958 14950 tablet_bootstrap.cc:492] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:43.117839 14950 tablet_bootstrap.cc:492] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:43.118497 14950 ts_tablet_manager.cc:1397] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.014s
I20260807 08:00:43.121371 14950 raft_consensus.cc:348] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:43.122359 14950 raft_consensus.cc:729] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:43.123039 14950 consensus_queue.cc:260] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:43.127230 14950 ts_tablet_manager.cc:1428] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260807 08:00:43.128846 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-25fcdcbbf735436893ce5c13fd2eb1f1 received from {username='slave'} at 127.0.0.1:45382
I20260807 08:00:43.129221 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-25fcdcbbf735436893ce5c13fd2eb1f1 on tablet 25fcdcbbf735436893ce5c13fd2eb1f1 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:43.130756 13608 tablet_copy_source_session.cc:205] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:43.134902 14948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e01d60f0064bb3958a65250e430990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:43.149156 14948 tablet_copy_client.cc:791] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:43.149729 14948 tablet_copy_client.cc:655] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:43.153002 14948 tablet_copy_client.cc:531] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:43.161602 14948 tablet_bootstrap.cc:492] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:43.178699 14948 tablet_bootstrap.cc:492] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:43.179592 14948 tablet_bootstrap.cc:492] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:43.180269 14948 ts_tablet_manager.cc:1397] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20260807 08:00:43.182928 14948 raft_consensus.cc:348] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:43.183720 14948 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:43.184386 14948 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:43.186453 14948 ts_tablet_manager.cc:1428] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260807 08:00:43.188182 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-e4e01d60f0064bb3958a65250e430990 received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:43.188553 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-e4e01d60f0064bb3958a65250e430990 on tablet e4e01d60f0064bb3958a65250e430990 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:43.365087 14630 raft_consensus.cc:1203] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:43.474752 14630 raft_consensus.cc:1203] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:43.511124 14630 raft_consensus.cc:1203] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:43.594096 14630 raft_consensus.cc:1203] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:43.865001 14804 raft_consensus.cc:1050] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:43.866928 14804 consensus_queue.cc:237] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:43.871959 14630 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:43.873155 14966 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:43.874081 13809 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:43.876194 14804 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:43.880540 13584 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:43.882697 14966 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:43.885542 14971 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:43.893019 13585 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:43.893098 13816 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:43.904251 14630 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:43.914324 13428 catalog_manager.cc:5495] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:43.955628 13703 consensus_queue.cc:237] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:43.960431 14630 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:43.962147 14804 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:43.971069 14933 raft_consensus.cc:1050] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:43.970474 14876 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:43.972257 14630 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:43.980444 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:43.979414 14933 consensus_queue.cc:237] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:43.991268 13431 catalog_manager.cc:5495] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:43.985591 13585 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:43.998675 14876 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:44.000442 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d5ddf3f7d84e0ebda02d99abff4f0e with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:44.015504 13816 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:44.017452 14630 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:44.017462 14933 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:44.018067 13711 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:44.020593 14934 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:44.022068 14985 tablet_replica.cc:330] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:00:44.022387 14965 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:44.023922 14985 raft_consensus.cc:2229] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:44.025355 14985 raft_consensus.cc:2258] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:44.030313 14804 raft_consensus.cc:1050] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:44.033166 14985 ts_tablet_manager.cc:1905] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:44.032711 14804 consensus_queue.cc:237] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:44.018518 13585 raft_consensus.cc:2932] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.091190 14965 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.094981 14630 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.112243 13430 catalog_manager.cc:5495] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.117774 13585 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:44.119580 14630 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:44.120224 14966 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:44.122109 14966 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:44.113754 14985 ts_tablet_manager.cc:1918] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:44.115310 13811 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.131938 13705 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.116662 13816 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:44.138515 14966 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:44.140336 14985 log.cc:1196] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/46d5ddf3f7d84e0ebda02d99abff4f0e
I20260807 08:00:44.149737 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet 46d5ddf3f7d84e0ebda02d99abff4f0e (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:44.172739 14966 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.175027 14630 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.184408 13816 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.191002 13430 catalog_manager.cc:5495] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.206213 13592 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.207969 13585 consensus_queue.cc:237] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:44.215291 14630 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:44.216558 14995 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:44.216991 13816 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:44.219043 14995 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260807 08:00:44.226919 14934 raft_consensus.cc:1050] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:44.226655 14630 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.229267 14934 consensus_queue.cc:237] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:44.225023 14995 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.232947 13705 consensus_queue.cc:237] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:44.240134 13816 raft_consensus.cc:2932] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.248188 14627 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:44.253686 14876 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:44.260015 13432 catalog_manager.cc:5495] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.261324 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 35cf7298a676441c8d12e04b066ee036 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:44.268436 13811 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:44.269230 14629 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:44.271296 14934 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:44.272149 13816 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:44.272394 14988 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:44.273597 14966 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:44.274874 13705 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:44.277406 14965 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:44.286319 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet 35cf7298a676441c8d12e04b066ee036 with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:00:44.285976 14992 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.295327 13809 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.300071 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 25fcdcbbf735436893ce5c13fd2eb1f1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:44.310941 15009 tablet_replica.cc:330] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:44.310851 14630 raft_consensus.cc:2932] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.315167 15009 raft_consensus.cc:2229] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:44.310070 13430 catalog_manager.cc:5495] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:44.319939 15009 raft_consensus.cc:2258] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:44.323087 15009 ts_tablet_manager.cc:1905] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:44.324656 14995 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.332085 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet 25fcdcbbf735436893ce5c13fd2eb1f1 with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:00:44.331396 14629 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.339908 15009 ts_tablet_manager.cc:1918] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:44.340340 15009 log.cc:1196] T 35cf7298a676441c8d12e04b066ee036 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/35cf7298a676441c8d12e04b066ee036
I20260807 08:00:44.342319 15010 tablet_replica.cc:330] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:00:44.343469 15010 raft_consensus.cc:2229] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:44.344085 15010 raft_consensus.cc:2258] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:44.344630 13708 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.347493 15010 ts_tablet_manager.cc:1905] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:44.350448 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet 35cf7298a676441c8d12e04b066ee036 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:44.362994 15010 ts_tablet_manager.cc:1918] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:44.363552 15010 log.cc:1196] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/25fcdcbbf735436893ce5c13fd2eb1f1
I20260807 08:00:44.366101 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet 25fcdcbbf735436893ce5c13fd2eb1f1 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:44.384262 13428 catalog_manager.cc:5495] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.388505 13809 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.411970 13584 consensus_queue.cc:237] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:44.430269 14630 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:44.431074 13708 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:44.434136 14934 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:44.436100 14934 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:44.441699 14965 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.443583 13708 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.445678 14630 raft_consensus.cc:2932] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:44.454231 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e4e01d60f0064bb3958a65250e430990 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:44.460090 13428 catalog_manager.cc:5495] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:44.469702 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e01d60f0064bb3958a65250e430990 with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:44.470589 14985 tablet_replica.cc:330] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:00:44.471652 14985 raft_consensus.cc:2229] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:44.472218 14985 raft_consensus.cc:2258] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:44.475111 14985 ts_tablet_manager.cc:1905] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:44.488768 14985 ts_tablet_manager.cc:1918] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:44.489252 14985 log.cc:1196] T e4e01d60f0064bb3958a65250e430990 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/e4e01d60f0064bb3958a65250e430990
I20260807 08:00:44.491062 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet e4e01d60f0064bb3958a65250e430990 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:45.518884 13592 consensus_queue.cc:237] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:45.526371 13809 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:45.527084 13540 consensus_peers.cc:487] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e37d72ff9b640289fad70dde36572fe. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:45.527401 13705 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:45.527521 15013 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:45.528610 15017 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:45.536517 14934 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.538713 13708 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.543231 13809 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.554553 13432 catalog_manager.cc:5495] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.566088 13809 consensus_queue.cc:237] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:45.574350 13705 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:45.574790 13769 consensus_peers.cc:487] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d37920c01cb94122bd1c5a0bb331be31. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:45.575289 13592 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:45.576725 14790 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:45.579403 15027 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:45.588757 14922 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.612279 13592 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.614492 13705 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.618304 13584 consensus_queue.cc:237] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:45.631721 13428 catalog_manager.cc:5495] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260807 08:00:45.639880 13540 consensus_peers.cc:487] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c9ed355e84f4fc7a80a1d59a76a365c. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:45.643317 13809 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:45.647364 13705 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:45.648388 15017 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:45.650015 15017 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:45.679996 14934 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.687563 13705 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.689564 13816 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.701922 13428 catalog_manager.cc:5495] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:45.704555 13809 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
W20260807 08:00:45.720019 13769 consensus_peers.cc:487] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a01b1964575c4708bf9760dff95db7f3. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:45.721748 13592 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:45.722810 13705 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:45.723116 15036 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:45.724037 15037 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:45.747543 15037 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.749380 13592 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.751590 13705 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.768486 13430 catalog_manager.cc:5495] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:45.991886 15049 ts_tablet_manager.cc:927] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:45.993747 15049 tablet_copy_client.cc:316] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:45.995759 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet d37920c01cb94122bd1c5a0bb331be31 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:45.996300 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet d37920c01cb94122bd1c5a0bb331be31 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-d37920c01cb94122bd1c5a0bb331be31
I20260807 08:00:46.003765 13830 tablet_copy_source_session.cc:205] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:46.006877 15049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37920c01cb94122bd1c5a0bb331be31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:46.020747 15049 tablet_copy_client.cc:791] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:46.021379 15049 tablet_copy_client.cc:655] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:46.025507 15049 tablet_copy_client.cc:531] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:46.032652 15049 tablet_bootstrap.cc:492] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:46.051921 15049 tablet_bootstrap.cc:492] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:46.052758 15049 tablet_bootstrap.cc:492] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:46.053282 15049 ts_tablet_manager.cc:1397] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.003s
I20260807 08:00:46.055456 15049 raft_consensus.cc:348] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:46.056103 15049 raft_consensus.cc:729] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:46.056732 15049 consensus_queue.cc:260] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:46.060443 15049 ts_tablet_manager.cc:1428] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260807 08:00:46.061269 15054 ts_tablet_manager.cc:927] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:46.063298 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-d37920c01cb94122bd1c5a0bb331be31 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:46.063727 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-d37920c01cb94122bd1c5a0bb331be31 on tablet d37920c01cb94122bd1c5a0bb331be31 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:46.066078 15054 tablet_copy_client.cc:316] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:46.067782 13608 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet 2e37d72ff9b640289fad70dde36572fe from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:46.068274 13608 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet 2e37d72ff9b640289fad70dde36572fe from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-2e37d72ff9b640289fad70dde36572fe
I20260807 08:00:46.079692 15056 ts_tablet_manager.cc:927] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:46.083201 15056 tablet_copy_client.cc:316] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:46.085083 13606 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet 4c9ed355e84f4fc7a80a1d59a76a365c from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:46.085721 13606 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet 4c9ed355e84f4fc7a80a1d59a76a365c from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-4c9ed355e84f4fc7a80a1d59a76a365c
I20260807 08:00:46.093461 13606 tablet_copy_source_session.cc:205] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:46.096570 13608 tablet_copy_source_session.cc:205] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:46.098068 15056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c9ed355e84f4fc7a80a1d59a76a365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:46.102020 15054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e37d72ff9b640289fad70dde36572fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:46.114928 15056 tablet_copy_client.cc:791] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:46.115590 15056 tablet_copy_client.cc:655] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:46.119266 15056 tablet_copy_client.cc:531] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:46.120509 15054 tablet_copy_client.cc:791] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:46.121124 15054 tablet_copy_client.cc:655] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:46.124886 15054 tablet_copy_client.cc:531] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:46.128441 15056 tablet_bootstrap.cc:492] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:46.130648 15049 ts_tablet_manager.cc:927] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:46.132300 15049 tablet_copy_client.cc:316] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:46.133970 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet a01b1964575c4708bf9760dff95db7f3 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:46.134500 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet a01b1964575c4708bf9760dff95db7f3 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-a01b1964575c4708bf9760dff95db7f3
I20260807 08:00:46.142959 13830 tablet_copy_source_session.cc:205] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:46.149461 15049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01b1964575c4708bf9760dff95db7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:46.156452 15056 tablet_bootstrap.cc:492] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:46.157585 15056 tablet_bootstrap.cc:492] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:46.158316 15056 ts_tablet_manager.cc:1397] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.008s
I20260807 08:00:46.160730 15054 tablet_bootstrap.cc:492] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:46.161948 15056 raft_consensus.cc:348] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:46.165858 15056 raft_consensus.cc:729] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:46.166561 15056 consensus_queue.cc:260] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:46.169260 15056 ts_tablet_manager.cc:1428] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260807 08:00:46.169894 15049 tablet_copy_client.cc:791] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:46.171141 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-4c9ed355e84f4fc7a80a1d59a76a365c received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:46.171144 15049 tablet_copy_client.cc:655] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:46.171711 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-4c9ed355e84f4fc7a80a1d59a76a365c on tablet 4c9ed355e84f4fc7a80a1d59a76a365c with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:46.182119 15049 tablet_copy_client.cc:531] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:46.193629 15049 tablet_bootstrap.cc:492] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:46.262300 15054 tablet_bootstrap.cc:492] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:46.262392 15049 tablet_bootstrap.cc:492] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:46.263296 15054 tablet_bootstrap.cc:492] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:46.263296 15049 tablet_bootstrap.cc:492] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:46.264163 15054 ts_tablet_manager.cc:1397] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.104s user 0.025s sys 0.042s
I20260807 08:00:46.264200 15049 ts_tablet_manager.cc:1397] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.071s user 0.036s sys 0.032s
I20260807 08:00:46.267197 15049 raft_consensus.cc:348] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:46.267411 15054 raft_consensus.cc:348] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:46.268168 15049 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:46.268249 15054 raft_consensus.cc:729] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:46.269062 15049 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:46.269140 15054 consensus_queue.cc:260] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:46.272496 15049 ts_tablet_manager.cc:1428] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260807 08:00:46.278095 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-a01b1964575c4708bf9760dff95db7f3 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:46.278573 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-a01b1964575c4708bf9760dff95db7f3 on tablet a01b1964575c4708bf9760dff95db7f3 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:46.288300 15054 ts_tablet_manager.cc:1428] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.024s user 0.016s sys 0.000s
I20260807 08:00:46.290122 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-2e37d72ff9b640289fad70dde36572fe received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:46.290542 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-2e37d72ff9b640289fad70dde36572fe on tablet 2e37d72ff9b640289fad70dde36572fe with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:46.476706 14630 raft_consensus.cc:1203] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:46.536998 14630 raft_consensus.cc:1203] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:46.593582 14630 raft_consensus.cc:1203] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:46.674360 14629 raft_consensus.cc:1203] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:47.052584 14934 raft_consensus.cc:1050] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:47.054405 14934 consensus_queue.cc:237] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:47.062002 13809 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.062350 14629 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.062551 13708 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.063977 15013 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.065285 14934 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.066082 15017 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.079282 15071 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.082213 14630 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.083109 13711 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.096153 13428 catalog_manager.cc:5495] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.105221 13811 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.114637 15017 raft_consensus.cc:1050] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:47.119165 15017 consensus_queue.cc:237] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:47.148650 13591 consensus_queue.cc:237] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:47.159462 14630 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:47.163699 13708 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:47.162891 13816 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.162693 15017 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260807 08:00:47.166015 15071 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.167382 15017 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:47.168237 14629 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.170950 15071 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.169436 13711 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.177260 15013 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.182494 14934 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.190976 15027 raft_consensus.cc:1050] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:47.193759 15027 consensus_queue.cc:237] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:47.196209 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2e37d72ff9b640289fad70dde36572fe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:47.202705 14630 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.208640 13705 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.210434 14922 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.211999 13711 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.216729 14922 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.225220 13591 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.232425 14934 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.233904 14629 raft_consensus.cc:2932] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.234258 14627 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.229000 13431 catalog_manager.cc:5495] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:47.257777 13430 catalog_manager.cc:5495] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.262679 15036 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:47.280622 13809 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.292936 13708 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.315623 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e37d72ff9b640289fad70dde36572fe with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:47.320953 15101 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.330821 15027 raft_consensus.cc:1050] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:47.333433 15027 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:47.335862 13585 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.344179 14629 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.350986 13591 consensus_queue.cc:237] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:47.354696 15103 tablet_replica.cc:330] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:00:47.361893 14630 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:47.363512 13809 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:47.365156 14934 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:47.366686 14934 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:47.366899 13430 catalog_manager.cc:5495] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.370573 15103 raft_consensus.cc:2229] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:47.371443 15103 raft_consensus.cc:2258] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:47.371001 13705 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.376236 15103 ts_tablet_manager.cc:1905] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:47.376430 15017 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.382171 13809 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.386484 14629 raft_consensus.cc:2932] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.396214 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4c9ed355e84f4fc7a80a1d59a76a365c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:47.400204 15103 ts_tablet_manager.cc:1918] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:47.400700 15103 log.cc:1196] T 2e37d72ff9b640289fad70dde36572fe P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/2e37d72ff9b640289fad70dde36572fe
I20260807 08:00:47.408262 14630 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.410127 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet 2e37d72ff9b640289fad70dde36572fe (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:47.410840 13705 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.416455 15106 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:47.418509 15106 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.418978 13809 consensus_queue.cc:237] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:47.426947 14627 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:47.429814 15105 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:00:47.430433 13711 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:47.431269 13584 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:47.432824 15105 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:47.428439 13431 catalog_manager.cc:5495] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.436218 15105 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:47.447748 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c9ed355e84f4fc7a80a1d59a76a365c with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:00:47.451622 15106 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.463560 13708 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.464668 14630 raft_consensus.cc:2932] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.450174 15120 tablet_replica.cc:330] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:47.477958 15105 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.483227 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d37920c01cb94122bd1c5a0bb331be31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:47.482893 13705 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.481994 15120 raft_consensus.cc:2229] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:47.491357 15120 raft_consensus.cc:2258] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:47.493556 13432 catalog_manager.cc:5495] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:47.502525 14629 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.509045 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet d37920c01cb94122bd1c5a0bb331be31 with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:00:47.517741 15120 ts_tablet_manager.cc:1905] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:47.520778 13430 catalog_manager.cc:5495] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.525924 13584 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.551230 15122 tablet_replica.cc:330] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:00:47.552816 15122 raft_consensus.cc:2229] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:47.553409 15120 ts_tablet_manager.cc:1918] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:47.553954 15120 log.cc:1196] T 4c9ed355e84f4fc7a80a1d59a76a365c P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/4c9ed355e84f4fc7a80a1d59a76a365c
I20260807 08:00:47.553578 15122 raft_consensus.cc:2258] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:47.555661 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet 4c9ed355e84f4fc7a80a1d59a76a365c (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:47.562773 15122 ts_tablet_manager.cc:1905] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:47.575749 13809 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:47.578115 15122 ts_tablet_manager.cc:1918] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:47.578619 15122 log.cc:1196] T d37920c01cb94122bd1c5a0bb331be31 P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/d37920c01cb94122bd1c5a0bb331be31
I20260807 08:00:47.581297 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet d37920c01cb94122bd1c5a0bb331be31 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:47.584277 14629 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:47.585799 15105 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:47.586074 13584 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:47.588470 15105 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:47.595008 15106 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.596737 14629 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.605105 13584 raft_consensus.cc:2932] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.610930 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a01b1964575c4708bf9760dff95db7f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:47.616053 13430 catalog_manager.cc:5495] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:47.629915 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet a01b1964575c4708bf9760dff95db7f3 with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:00:47.630762 15120 tablet_replica.cc:330] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:47.631731 15120 raft_consensus.cc:2229] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:47.632475 15120 raft_consensus.cc:2258] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:47.637398 15120 ts_tablet_manager.cc:1905] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:47.656663 15120 ts_tablet_manager.cc:1918] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:47.657058 15120 log.cc:1196] T a01b1964575c4708bf9760dff95db7f3 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/a01b1964575c4708bf9760dff95db7f3
I20260807 08:00:47.658681 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet a01b1964575c4708bf9760dff95db7f3 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:48.691107 13809 consensus_queue.cc:237] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:48.697530 13705 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:48.697826 13584 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:48.699056 15036 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 08:00:48.700093 13769 consensus_peers.cc:487] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a8da02e8f51481cbcca504f8cff6f80. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:48.700132 15106 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:48.708144 13592 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.709573 13705 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.706739 15036 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.724822 13430 catalog_manager.cc:5495] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.734922 13592 consensus_queue.cc:237] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:48.754874 13809 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:48.755604 13705 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:48.756598 13540 consensus_peers.cc:487] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c78744cb93ef47e9aacb62cbeb454457. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:48.757252 15125 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:48.759634 15125 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:48.769452 15125 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.771288 13705 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.794634 13809 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.796022 13431 catalog_manager.cc:5495] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.819000 13809 consensus_queue.cc:237] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
W20260807 08:00:48.825807 13769 consensus_peers.cc:487] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b4b1bd6a74d452e860b1420b0ef1434. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:48.826479 13592 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:48.826897 13705 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:48.828135 15106 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:48.829385 15136 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:48.838249 15106 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.840029 13592 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.840847 13705 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.852807 13584 consensus_queue.cc:237] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:48.859990 13431 catalog_manager.cc:5495] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:48.871812 13809 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:48.873260 13540 consensus_peers.cc:487] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74ccb94230614cc48e846580ad82e994. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:48.874018 13705 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:48.875339 15125 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:48.877163 15144 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:48.902674 15148 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.904253 13809 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.908818 13705 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:48.949846 13430 catalog_manager.cc:5495] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:49.246234 15165 ts_tablet_manager.cc:927] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:49.248229 15165 tablet_copy_client.cc:316] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:49.249662 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet 8a8da02e8f51481cbcca504f8cff6f80 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:49.250190 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet 8a8da02e8f51481cbcca504f8cff6f80 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-8a8da02e8f51481cbcca504f8cff6f80
I20260807 08:00:49.254876 15167 ts_tablet_manager.cc:927] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:49.263761 13830 tablet_copy_source_session.cc:205] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:49.268043 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8da02e8f51481cbcca504f8cff6f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:49.266813 15167 tablet_copy_client.cc:316] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:49.279002 13608 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet 74ccb94230614cc48e846580ad82e994 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:49.279443 13608 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet 74ccb94230614cc48e846580ad82e994 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-74ccb94230614cc48e846580ad82e994
I20260807 08:00:49.285269 13608 tablet_copy_source_session.cc:205] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:49.286440 15169 ts_tablet_manager.cc:927] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:49.290977 15165 tablet_copy_client.cc:791] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:49.290722 15167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ccb94230614cc48e846580ad82e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:49.291533 15165 tablet_copy_client.cc:655] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:49.297624 15165 tablet_copy_client.cc:531] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:49.301904 15169 tablet_copy_client.cc:316] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:49.303570 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet 9b4b1bd6a74d452e860b1420b0ef1434 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:49.304061 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet 9b4b1bd6a74d452e860b1420b0ef1434 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-9b4b1bd6a74d452e860b1420b0ef1434
I20260807 08:00:49.306417 15165 tablet_bootstrap.cc:492] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:49.312592 13830 tablet_copy_source_session.cc:205] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:49.314375 15171 ts_tablet_manager.cc:927] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:49.317302 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4b1bd6a74d452e860b1420b0ef1434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:49.318692 15171 tablet_copy_client.cc:316] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:49.320098 13608 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet c78744cb93ef47e9aacb62cbeb454457 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:49.320588 13608 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet c78744cb93ef47e9aacb62cbeb454457 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-c78744cb93ef47e9aacb62cbeb454457
I20260807 08:00:49.343334 15167 tablet_copy_client.cc:791] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:49.343938 15167 tablet_copy_client.cc:655] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:49.344619 13608 tablet_copy_source_session.cc:205] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:49.349617 15167 tablet_copy_client.cc:531] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:49.350559 15171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78744cb93ef47e9aacb62cbeb454457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:49.355525 15169 tablet_copy_client.cc:791] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:49.356055 15169 tablet_copy_client.cc:655] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:49.359190 15167 tablet_bootstrap.cc:492] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:49.362517 15169 tablet_copy_client.cc:531] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:49.365336 15165 tablet_bootstrap.cc:492] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:49.368754 15165 tablet_bootstrap.cc:492] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:49.369500 15165 ts_tablet_manager.cc:1397] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.017s
I20260807 08:00:49.372303 15165 raft_consensus.cc:348] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:49.373195 15171 tablet_copy_client.cc:791] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:49.373880 15171 tablet_copy_client.cc:655] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:49.374361 15165 raft_consensus.cc:729] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:49.375097 15165 consensus_queue.cc:260] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:49.382063 15171 tablet_copy_client.cc:531] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:49.383543 15169 tablet_bootstrap.cc:492] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:49.391640 15167 tablet_bootstrap.cc:492] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:49.394614 15167 tablet_bootstrap.cc:492] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:49.395323 15167 ts_tablet_manager.cc:1397] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.036s user 0.023s sys 0.004s
I20260807 08:00:49.398219 15167 raft_consensus.cc:348] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:49.399125 15167 raft_consensus.cc:729] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:49.399915 15167 consensus_queue.cc:260] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:49.406046 15171 tablet_bootstrap.cc:492] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:49.411326 15167 ts_tablet_manager.cc:1428] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.016s user 0.005s sys 0.008s
I20260807 08:00:49.413929 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-74ccb94230614cc48e846580ad82e994 received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:49.414321 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-74ccb94230614cc48e846580ad82e994 on tablet 74ccb94230614cc48e846580ad82e994 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:49.422284 15169 tablet_bootstrap.cc:492] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:49.423265 15169 tablet_bootstrap.cc:492] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:49.423336 15165 ts_tablet_manager.cc:1428] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.053s user 0.009s sys 0.003s
I20260807 08:00:49.423971 15169 ts_tablet_manager.cc:1397] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.041s user 0.016s sys 0.011s
I20260807 08:00:49.426067 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-8a8da02e8f51481cbcca504f8cff6f80 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:49.426451 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-8a8da02e8f51481cbcca504f8cff6f80 on tablet 8a8da02e8f51481cbcca504f8cff6f80 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:49.433341 15169 raft_consensus.cc:348] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:49.435698 15169 raft_consensus.cc:729] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:49.436455 15169 consensus_queue.cc:260] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:49.438858 15169 ts_tablet_manager.cc:1428] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260807 08:00:49.445802 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-9b4b1bd6a74d452e860b1420b0ef1434 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:49.446188 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-9b4b1bd6a74d452e860b1420b0ef1434 on tablet 9b4b1bd6a74d452e860b1420b0ef1434 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:49.461122 15171 tablet_bootstrap.cc:492] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:49.467077 15171 tablet_bootstrap.cc:492] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:49.468004 15171 ts_tablet_manager.cc:1397] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.062s user 0.017s sys 0.005s
I20260807 08:00:49.470628 15171 raft_consensus.cc:348] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:49.471239 15171 raft_consensus.cc:729] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:49.471731 15171 consensus_queue.cc:260] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:49.473551 15171 ts_tablet_manager.cc:1428] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260807 08:00:49.475099 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-c78744cb93ef47e9aacb62cbeb454457 received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:49.475503 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-c78744cb93ef47e9aacb62cbeb454457 on tablet c78744cb93ef47e9aacb62cbeb454457 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:49.670135 14630 raft_consensus.cc:1203] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:49.770788 14629 raft_consensus.cc:1203] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:49.806344 14629 raft_consensus.cc:1203] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:49.843458 14629 raft_consensus.cc:1203] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:50.161077 15185 raft_consensus.cc:1050] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:50.163229 15185 consensus_queue.cc:237] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:50.168938 14629 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.169217 13711 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.170451 13591 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.170944 15126 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:50.172154 15185 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:50.173290 15163 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:50.187410 15148 raft_consensus.cc:1050] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:50.193348 15148 consensus_queue.cc:237] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:50.196451 15163 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.197907 13711 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.202363 13589 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.216508 13430 catalog_manager.cc:5495] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.232210 14629 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.243726 13816 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.245754 15125 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:50.252372 14630 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.255167 13711 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.256597 15125 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:50.259063 15017 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:50.286800 15125 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.293457 14629 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.289626 13711 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.312819 15148 raft_consensus.cc:1050] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:50.315147 15148 consensus_queue.cc:237] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:50.314474 13430 catalog_manager.cc:5495] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.316906 13816 consensus_queue.cc:237] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:50.332700 14629 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.332947 13816 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.339427 13711 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:50.340108 15017 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:50.342293 14629 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:50.352360 13711 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.353430 15144 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:50.359794 13589 consensus_queue.cc:237] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:50.360911 15144 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:50.365787 14630 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:50.367609 13711 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:50.371654 15200 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:00:50.374789 15144 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:50.380565 15200 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.381232 14630 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.382825 15163 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:50.385594 13711 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.392349 15163 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:00:50.392071 13703 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.398779 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 74ccb94230614cc48e846580ad82e994 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:50.402024 13430 catalog_manager.cc:5495] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.390257 15125 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.430768 13432 catalog_manager.cc:5495] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.438324 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ccb94230614cc48e846580ad82e994 with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
W20260807 08:00:50.438798 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 104629us
I20260807 08:00:50.442952 15036 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.443866 14630 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.449448 13816 raft_consensus.cc:2932] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.466387 15185 raft_consensus.cc:1050] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:50.468902 15185 consensus_queue.cc:237] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:50.475590 13811 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.490932 15209 tablet_replica.cc:330] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:00:50.491964 15209 raft_consensus.cc:2229] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:50.492666 15209 raft_consensus.cc:2258] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 08:00:50.492751 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 51953.2us
I20260807 08:00:50.497315 15209 ts_tablet_manager.cc:1905] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:50.498277 14630 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.499796 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9b4b1bd6a74d452e860b1420b0ef1434 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:50.506494 13589 consensus_queue.cc:237] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:50.508116 14629 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.507565 13430 catalog_manager.cc:5495] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:50.508757 13711 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.510879 13711 raft_consensus.cc:2932] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.517199 13708 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:50.517725 13589 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:50.517825 14629 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:50.518999 15203 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:50.521247 15203 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:50.526458 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b4b1bd6a74d452e860b1420b0ef1434 with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:00:50.528242 15214 tablet_replica.cc:330] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:00:50.529275 15214 raft_consensus.cc:2229] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:50.530697 15203 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.531657 13708 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.533776 15036 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:50.534257 14629 raft_consensus.cc:2932] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.548270 15163 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:50.551360 15214 raft_consensus.cc:2258] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:50.551522 15209 ts_tablet_manager.cc:1918] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:50.551985 15209 log.cc:1196] T 74ccb94230614cc48e846580ad82e994 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/74ccb94230614cc48e846580ad82e994
I20260807 08:00:50.554558 15214 ts_tablet_manager.cc:1905] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:50.554735 15126 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:50.555680 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c78744cb93ef47e9aacb62cbeb454457 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:50.566895 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet 74ccb94230614cc48e846580ad82e994 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:50.580406 13430 catalog_manager.cc:5495] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.674965 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet c78744cb93ef47e9aacb62cbeb454457 with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:50.678992 15214 ts_tablet_manager.cc:1918] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:50.679488 15214 log.cc:1196] T 9b4b1bd6a74d452e860b1420b0ef1434 P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/9b4b1bd6a74d452e860b1420b0ef1434
I20260807 08:00:50.683166 15209 tablet_replica.cc:330] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:00:50.688040 15209 raft_consensus.cc:2229] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:50.688702 15209 raft_consensus.cc:2258] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:50.693885 15209 ts_tablet_manager.cc:1905] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:50.706353 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet 9b4b1bd6a74d452e860b1420b0ef1434 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:50.711333 15163 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.715327 13589 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.723453 14627 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.719815 13708 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.736658 13430 catalog_manager.cc:5495] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.739292 15209 ts_tablet_manager.cc:1918] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:50.761957 15209 log.cc:1196] T c78744cb93ef47e9aacb62cbeb454457 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/c78744cb93ef47e9aacb62cbeb454457
I20260807 08:00:50.767411 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet c78744cb93ef47e9aacb62cbeb454457 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:50.774951 13811 consensus_queue.cc:237] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:50.781915 14627 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:50.781988 13708 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:50.783236 15163 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:50.784127 15216 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:50.790867 15185 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.796988 13708 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.801019 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8a8da02e8f51481cbcca504f8cff6f80 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:50.801826 14627 raft_consensus.cc:2932] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.818943 13432 catalog_manager.cc:5495] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:50.830109 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8da02e8f51481cbcca504f8cff6f80 with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:00:50.831128 15214 tablet_replica.cc:330] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:00:50.832222 15214 raft_consensus.cc:2229] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:50.832839 15214 raft_consensus.cc:2258] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:50.835807 15214 ts_tablet_manager.cc:1905] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:50.850750 15214 ts_tablet_manager.cc:1918] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:50.851133 15214 log.cc:1196] T 8a8da02e8f51481cbcca504f8cff6f80 P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/8a8da02e8f51481cbcca504f8cff6f80
I20260807 08:00:50.852540 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet 8a8da02e8f51481cbcca504f8cff6f80 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:51.883450 13811 consensus_queue.cc:237] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:51.895533 13708 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:51.895875 13591 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:51.896840 13769 consensus_peers.cc:487] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b74446812e814c7e9b4a1bd80bfb485d. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:51.897693 15224 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:51.898592 15185 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:51.907683 15224 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:51.910805 13591 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:51.911553 13708 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:51.922657 13711 consensus_queue.cc:237] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:51.928354 13431 catalog_manager.cc:5495] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:51.930064 13811 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:51.930943 13661 consensus_peers.cc:487] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2dfc89fb081405f96a0b53d76aeea38. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:51.931505 15159 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:51.942617 13591 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:51.946802 15124 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:51.953235 13816 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:51.951658 15159 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:51.976917 13591 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:51.977331 13816 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:51.999514 13432 catalog_manager.cc:5495] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:52.003839 13708 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:52.007480 15185 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 08:00:52.011904 13769 consensus_peers.cc:487] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04e678279be64e6ca8a141d6b761e522. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:52.017261 13591 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:52.018795 15226 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:52.021011 15246 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:52.023926 13708 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:52.028101 13591 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:52.036959 13711 consensus_queue.cc:237] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:52.039172 13431 catalog_manager.cc:5495] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:52.053563 13816 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:52.055094 13661 consensus_peers.cc:487] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ab8c56441f84d86b5798c528463965c. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:52.055392 15159 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:52.055876 13591 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:52.057214 15159 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:52.071703 15222 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:52.073913 13816 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:52.086498 13591 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:52.094022 13431 catalog_manager.cc:5495] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:52.366238 15262 ts_tablet_manager.cc:927] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:52.379835 15262 tablet_copy_client.cc:316] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:52.384605 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet b74446812e814c7e9b4a1bd80bfb485d from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:52.385468 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet b74446812e814c7e9b4a1bd80bfb485d from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-b74446812e814c7e9b4a1bd80bfb485d
I20260807 08:00:52.419335 13830 tablet_copy_source_session.cc:205] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:52.427501 15262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74446812e814c7e9b4a1bd80bfb485d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:52.451756 15267 ts_tablet_manager.cc:927] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:52.453954 15267 tablet_copy_client.cc:316] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:52.455765 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet 04e678279be64e6ca8a141d6b761e522 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:52.456496 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet 04e678279be64e6ca8a141d6b761e522 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-04e678279be64e6ca8a141d6b761e522
I20260807 08:00:52.464923 15262 tablet_copy_client.cc:791] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:52.465596 15262 tablet_copy_client.cc:655] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:52.472117 15262 tablet_copy_client.cc:531] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:52.480273 13830 tablet_copy_source_session.cc:205] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:52.482587 15262 tablet_bootstrap.cc:492] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:52.485339 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e678279be64e6ca8a141d6b761e522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:52.541111 15273 ts_tablet_manager.cc:927] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:52.547474 15267 tablet_copy_client.cc:791] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:52.548269 15267 tablet_copy_client.cc:655] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:52.550858 15273 tablet_copy_client.cc:316] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:00:52.552969 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet d2dfc89fb081405f96a0b53d76aeea38 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:00:52.553740 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet d2dfc89fb081405f96a0b53d76aeea38 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-d2dfc89fb081405f96a0b53d76aeea38
I20260807 08:00:52.559181 15262 tablet_bootstrap.cc:492] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:52.560436 15262 tablet_bootstrap.cc:492] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:52.561314 15262 ts_tablet_manager.cc:1397] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.079s user 0.024s sys 0.004s
I20260807 08:00:52.564159 13724 tablet_copy_source_session.cc:205] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:52.564615 15267 tablet_copy_client.cc:531] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:52.568167 15273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2dfc89fb081405f96a0b53d76aeea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:52.573419 15267 tablet_bootstrap.cc:492] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:52.578214 15262 raft_consensus.cc:348] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:52.579926 15262 raft_consensus.cc:729] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:52.580801 15262 consensus_queue.cc:260] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:52.586431 15262 ts_tablet_manager.cc:1428] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.025s user 0.006s sys 0.003s
I20260807 08:00:52.589828 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-b74446812e814c7e9b4a1bd80bfb485d received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:52.590312 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-b74446812e814c7e9b4a1bd80bfb485d on tablet b74446812e814c7e9b4a1bd80bfb485d with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:52.609908 15273 tablet_copy_client.cc:791] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:52.610519 15273 tablet_copy_client.cc:655] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:52.614797 15273 tablet_copy_client.cc:531] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:52.610458 15262 ts_tablet_manager.cc:927] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:52.630264 15273 tablet_bootstrap.cc:492] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:52.634934 15262 tablet_copy_client.cc:316] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:00:52.643687 15267 tablet_bootstrap.cc:492] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:52.644629 15267 tablet_bootstrap.cc:492] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:52.645347 15267 ts_tablet_manager.cc:1397] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.072s user 0.018s sys 0.010s
I20260807 08:00:52.648933 15267 raft_consensus.cc:348] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:52.649827 15267 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:52.650667 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet 9ab8c56441f84d86b5798c528463965c from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:00:52.650493 15267 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:52.651228 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet 9ab8c56441f84d86b5798c528463965c from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-9ab8c56441f84d86b5798c528463965c
I20260807 08:00:52.652889 15273 tablet_bootstrap.cc:492] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:52.653545 15267 ts_tablet_manager.cc:1428] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260807 08:00:52.655140 15273 tablet_bootstrap.cc:492] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:52.655807 15273 ts_tablet_manager.cc:1397] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.000s
I20260807 08:00:52.655884 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-04e678279be64e6ca8a141d6b761e522 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:52.656302 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-04e678279be64e6ca8a141d6b761e522 on tablet 04e678279be64e6ca8a141d6b761e522 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:52.660498 15273 raft_consensus.cc:348] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:52.661396 15273 raft_consensus.cc:729] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:52.662009 15273 consensus_queue.cc:260] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:52.668143 13724 tablet_copy_source_session.cc:205] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:52.672294 15273 ts_tablet_manager.cc:1428] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.016s user 0.006s sys 0.007s
I20260807 08:00:52.674815 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-d2dfc89fb081405f96a0b53d76aeea38 received from {username='slave'} at 127.0.0.1:45382
I20260807 08:00:52.674983 15262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab8c56441f84d86b5798c528463965c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:52.675166 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-d2dfc89fb081405f96a0b53d76aeea38 on tablet d2dfc89fb081405f96a0b53d76aeea38 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:52.691461 15262 tablet_copy_client.cc:791] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:52.692006 15262 tablet_copy_client.cc:655] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:52.703356 15262 tablet_copy_client.cc:531] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:52.711464 15262 tablet_bootstrap.cc:492] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:52.732097 15262 tablet_bootstrap.cc:492] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:52.732929 15262 tablet_bootstrap.cc:492] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:52.733560 15262 ts_tablet_manager.cc:1397] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.001s
I20260807 08:00:52.735975 15262 raft_consensus.cc:348] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:52.736572 15262 raft_consensus.cc:729] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:52.737186 15262 consensus_queue.cc:260] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:52.739598 15262 ts_tablet_manager.cc:1428] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 08:00:52.741992 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-9ab8c56441f84d86b5798c528463965c received from {username='slave'} at 127.0.0.1:45382
I20260807 08:00:52.742405 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-9ab8c56441f84d86b5798c528463965c on tablet 9ab8c56441f84d86b5798c528463965c with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:52.851838 14630 raft_consensus.cc:1203] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:52.918488 14630 raft_consensus.cc:1203] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:53.034261 14630 raft_consensus.cc:1203] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:53.054548 14630 raft_consensus.cc:1203] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:53.315554 15246 raft_consensus.cc:1050] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:53.317968 15246 consensus_queue.cc:237] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:53.325203 14630 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.325848 13589 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.326702 15185 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:53.327270 13708 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.328536 15185 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:53.329566 15226 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:53.340626 15261 raft_consensus.cc:1050] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:53.343405 15261 consensus_queue.cc:237] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:53.345253 15226 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.348351 14630 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.356897 13589 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.361821 13703 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.367326 13431 catalog_manager.cc:5495] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.377117 13816 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.378574 15258 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:53.392140 13589 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.392204 14630 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.394205 15222 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:53.395320 15261 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:53.409246 15222 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.418130 14630 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.428076 13816 consensus_queue.cc:237] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:53.439314 14630 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:53.440280 15258 raft_consensus.cc:1050] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:53.440467 13592 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:53.442353 15185 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:53.443961 15185 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:53.444751 15258 consensus_queue.cc:237] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:53.448856 15226 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.450285 13432 catalog_manager.cc:5495] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.455493 13591 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.470034 14627 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.470293 13589 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.472487 13807 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.475245 15306 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:53.476594 15222 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:53.477495 15306 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260807 08:00:53.453961 14630 raft_consensus.cc:2932] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.486121 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b74446812e814c7e9b4a1bd80bfb485d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:53.457582 13811 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.491029 13589 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.507951 13432 catalog_manager.cc:5495] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:53.542030 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet b74446812e814c7e9b4a1bd80bfb485d with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:00:53.557871 15314 tablet_replica.cc:330] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:53.557675 13708 consensus_queue.cc:237] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:53.567166 14626 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:53.570250 15305 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:00:53.576750 15306 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.578085 13592 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:53.577976 13807 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.581772 15261 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:53.592901 14626 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.594583 13592 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.598412 15314 raft_consensus.cc:2229] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:53.599355 15314 raft_consensus.cc:2258] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:53.603921 13430 catalog_manager.cc:5495] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:53.605304 13591 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.605770 14630 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.610984 15314 ts_tablet_manager.cc:1905] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:53.609112 15306 raft_consensus.cc:2932] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.623246 13703 consensus_queue.cc:237] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:53.627089 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d2dfc89fb081405f96a0b53d76aeea38 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:53.633152 15314 ts_tablet_manager.cc:1918] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:53.633806 15314 log.cc:1196] T b74446812e814c7e9b4a1bd80bfb485d P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/b74446812e814c7e9b4a1bd80bfb485d
I20260807 08:00:53.640782 14625 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:53.643288 13807 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:53.638341 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet b74446812e814c7e9b4a1bd80bfb485d (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:53.647881 13431 catalog_manager.cc:5495] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.652447 15185 raft_consensus.cc:1050] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:53.652609 15222 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:53.655737 15185 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:53.660012 15305 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:53.674938 15261 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.679335 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet d2dfc89fb081405f96a0b53d76aeea38 with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:53.684854 15318 tablet_replica.cc:330] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:00:53.685971 15318 raft_consensus.cc:2229] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:53.686620 15318 raft_consensus.cc:2258] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:53.695441 15318 ts_tablet_manager.cc:1905] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:53.695359 13807 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.699194 14626 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.702216 13591 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:53.701995 15317 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:53.704262 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9ab8c56441f84d86b5798c528463965c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:53.704625 15317 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:53.706378 14626 raft_consensus.cc:2932] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.724907 13431 catalog_manager.cc:5495] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:53.744655 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab8c56441f84d86b5798c528463965c with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:00:53.747485 15246 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.749085 13591 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.757275 13708 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.772202 14626 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.773329 15318 ts_tablet_manager.cc:1918] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:53.773816 15318 log.cc:1196] T d2dfc89fb081405f96a0b53d76aeea38 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/d2dfc89fb081405f96a0b53d76aeea38
I20260807 08:00:53.773715 13431 catalog_manager.cc:5495] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.775547 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet d2dfc89fb081405f96a0b53d76aeea38 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:53.800323 15325 tablet_replica.cc:330] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:00:53.807868 13807 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:53.811179 15325 raft_consensus.cc:2229] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:53.812347 15325 raft_consensus.cc:2258] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:53.813834 13591 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:53.814672 14626 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:53.815378 15185 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:53.815719 15325 ts_tablet_manager.cc:1905] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:53.816195 15246 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:53.824995 15319 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.830125 13591 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.830492 14626 raft_consensus.cc:2932] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:53.836539 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 04e678279be64e6ca8a141d6b761e522 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:53.842440 15325 ts_tablet_manager.cc:1918] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:53.843222 15325 log.cc:1196] T 9ab8c56441f84d86b5798c528463965c P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/9ab8c56441f84d86b5798c528463965c
I20260807 08:00:53.845299 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet 9ab8c56441f84d86b5798c528463965c (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:53.847960 13428 catalog_manager.cc:5495] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:53.861132 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e678279be64e6ca8a141d6b761e522 with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:00:53.862067 15314 tablet_replica.cc:330] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:53.863080 15314 raft_consensus.cc:2229] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:53.863758 15314 raft_consensus.cc:2258] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:53.868770 15314 ts_tablet_manager.cc:1905] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:53.881455 15314 ts_tablet_manager.cc:1918] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:53.881871 15314 log.cc:1196] T 04e678279be64e6ca8a141d6b761e522 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/04e678279be64e6ca8a141d6b761e522
I20260807 08:00:53.883231 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet 04e678279be64e6ca8a141d6b761e522 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:54.463418 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:54.553761 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:00:54.554491 13811 raft_consensus.cc:595] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:54.569129 13703 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:00:54.569702 13703 raft_consensus.cc:595] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:54.571936 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:00:54.572338 13811 raft_consensus.cc:595] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:54.574298 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:00:54.574734 13811 raft_consensus.cc:595] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:54.576610 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:00:54.576982 13811 raft_consensus.cc:595] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:54.578768 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:00:54.579281 13811 raft_consensus.cc:595] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:54.580289 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260807 08:00:54.581816 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:54.582612 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:54.585923 15334 raft_consensus.cc:979] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:54.586421 15246 raft_consensus.cc:1067] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:54.587927 13591 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56594
I20260807 08:00:54.588436 13591 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:54.588764 13591 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.596676 13591 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:54.601068 13591 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:54.601893 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:54.602546 13703 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.603574 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:54.604068 13811 raft_consensus.cc:3032] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:54.604368 13811 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:54.605043 13811 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:54.606305 13811 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.610001 13703 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:54.611414 13540 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:54.612179 15333 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:54.612619 15333 raft_consensus.cc:686] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:54.612654 13811 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:54.613572 15333 consensus_queue.cc:237] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:54.631515 13431 catalog_manager.cc:5495] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:54.740430 15261 raft_consensus.cc:979] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:54.741003 15261 raft_consensus.cc:1067] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:54.741849 15334 raft_consensus.cc:979] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:54.742441 15246 raft_consensus.cc:1067] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:54.742480 13589 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56592
I20260807 08:00:54.743070 13589 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:54.743412 13589 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.743935 14626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42248
I20260807 08:00:54.744479 14626 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:54.744802 14626 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.750460 13589 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:54.752382 13589 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:54.753677 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:54.754264 13703 raft_consensus.cc:3032] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:54.753750 14626 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:54.754585 13703 raft_consensus.cc:729] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:54.754477 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:54.755296 13811 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.755501 13703 consensus_queue.cc:260] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:54.756724 13703 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.758057 14626 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:54.758249 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:54.758834 13807 raft_consensus.cc:3032] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:54.759119 13807 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:54.759392 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:54.759965 13589 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.759783 13807 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:54.760991 13807 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.762943 13811 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:54.767338 13589 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:54.767961 13703 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:54.765472 13537 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:54.770004 14583 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:00:54.771163 15333 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:54.771651 15333 raft_consensus.cc:686] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:54.771749 15342 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:54.772522 15333 consensus_queue.cc:237] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:54.777376 15342 raft_consensus.cc:686] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:54.778559 15342 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:54.792538 13431 catalog_manager.cc:5495] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:54.804159 13432 catalog_manager.cc:5495] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:54.809717 15319 raft_consensus.cc:979] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:54.810248 15334 raft_consensus.cc:1067] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:54.813333 14626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42248
I20260807 08:00:54.813941 14626 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:54.814337 14626 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.816985 13807 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:54.822995 14626 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:54.826018 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:54.826041 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:54.826730 13589 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.827175 13811 raft_consensus.cc:3032] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:54.834127 13811 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:54.834859 13811 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:54.835654 13589 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:54.836042 13811 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:54.836737 14583 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:00:54.836711 14626 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:54.837761 15342 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:54.840672 15342 raft_consensus.cc:686] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:54.842712 15342 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:54.846648 13811 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:54.859620 13432 catalog_manager.cc:5495] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:54.925006 13816 consensus_queue.cc:237] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:54.931103 13703 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:54.932065 13769 consensus_peers.cc:487] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc32413495ee41909c7c5494c8095084. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:54.932878 15246 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:54.933166 13589 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:54.934415 15246 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:54.940364 15334 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:54.941815 13703 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:54.944137 13589 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:54.954756 13591 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:54.957479 13432 catalog_manager.cc:5495] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:54.961961 13816 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:54.963973 15265 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 08:00:54.965425 13540 consensus_peers.cc:487] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 466eca488bfd44d396c22e2396d1ef7e. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:54.973016 13703 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:54.974825 15333 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:54.979171 15265 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:54.981155 13816 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:54.996208 13811 consensus_queue.cc:237] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.022475 13708 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:55.024986 13769 consensus_peers.cc:487] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 232127da9f574f41a5a8087ae775ef6b. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:55.030975 15319 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:55.035353 13432 catalog_manager.cc:5495] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:55.040635 13589 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:55.040367 13705 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:55.056016 15246 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:55.057050 15317 raft_consensus.cc:979] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:55.057585 15317 raft_consensus.cc:1067] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:55.059687 14626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42248
I20260807 08:00:55.060336 14626 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:55.060743 14626 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.071384 14626 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:55.074146 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:55.074509 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:55.074795 13589 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.075196 13811 raft_consensus.cc:3032] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:55.075553 13811 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:55.076304 13811 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:55.077497 13811 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.083513 13589 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:55.084457 13811 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:55.086810 14580 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:55.086036 15334 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:55.087746 15342 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:55.088299 15342 raft_consensus.cc:686] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:55.089303 15342 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:55.099110 13708 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:55.106520 13592 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:55.107092 13591 consensus_queue.cc:237] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.115404 13432 catalog_manager.cc:5495] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:55.126008 14626 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:55.127641 15317 raft_consensus.cc:979] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:55.128144 15246 raft_consensus.cc:1067] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:55.131346 13592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56594
I20260807 08:00:55.131811 13592 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:55.132126 13592 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.132738 13431 catalog_manager.cc:5495] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:55.140779 13592 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:55.143477 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:55.144009 13811 raft_consensus.cc:3032] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:55.144272 13811 raft_consensus.cc:729] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:55.144913 13811 consensus_queue.cc:260] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:55.145079 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:55.145638 13592 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:55.145776 13708 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.145862 13811 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.152642 13816 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:55.153077 13540 consensus_peers.cc:487] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5ef275b7c0a45619cbc05266aea24dc. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:55.154016 13811 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:55.154204 15333 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:55.155055 13537 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:55.155798 15364 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:55.156342 15364 raft_consensus.cc:686] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:55.156867 13705 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:55.157286 15364 consensus_queue.cc:237] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:55.162276 15265 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:55.174335 15333 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:55.180981 13811 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:55.187000 13431 catalog_manager.cc:5495] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:55.189695 13708 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:55.193997 13711 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:55.197414 15265 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:55.206022 13711 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:55.207055 13703 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:55.215612 15265 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:55.218262 13807 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:55.221346 13431 catalog_manager.cc:5495] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:55.219929 15333 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260807 08:00:55.252852 13816 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:55.254463 15326 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:55.266033 13589 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:55.270325 15342 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:00:55.275756 13816 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:55.277261 15342 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260807 08:00:55.315284 15386 log.cc:924] Time spent T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260807 08:00:55.352383 13807 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:55.355089 15342 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:55.368889 13592 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:55.375124 15343 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:00:55.432888 15397 ts_tablet_manager.cc:927] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:55.435341 15397 tablet_copy_client.cc:316] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:55.436932 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet cc32413495ee41909c7c5494c8095084 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:55.437503 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet cc32413495ee41909c7c5494c8095084 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-cc32413495ee41909c7c5494c8095084
I20260807 08:00:55.446412 13830 tablet_copy_source_session.cc:205] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:55.449592 15397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc32413495ee41909c7c5494c8095084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:55.464428 15397 tablet_copy_client.cc:791] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:55.465075 15397 tablet_copy_client.cc:655] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:55.469050 15397 tablet_copy_client.cc:531] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:55.477147 15397 tablet_bootstrap.cc:492] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:55.498303 15397 tablet_bootstrap.cc:492] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:55.499275 15397 tablet_bootstrap.cc:492] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:55.499938 15397 ts_tablet_manager.cc:1397] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20260807 08:00:55.502635 15397 raft_consensus.cc:348] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.503628 15397 raft_consensus.cc:729] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:55.504311 15397 consensus_queue.cc:260] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.507102 15397 ts_tablet_manager.cc:1428] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 08:00:55.509610 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-cc32413495ee41909c7c5494c8095084 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:55.510082 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-cc32413495ee41909c7c5494c8095084 on tablet cc32413495ee41909c7c5494c8095084 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:55.536438 13591 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:55.537673 15343 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:55.553361 15397 ts_tablet_manager.cc:927] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:55.555733 15397 tablet_copy_client.cc:316] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:55.562126 13608 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet e5ef275b7c0a45619cbc05266aea24dc from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:55.562793 13608 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet e5ef275b7c0a45619cbc05266aea24dc from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-e5ef275b7c0a45619cbc05266aea24dc
I20260807 08:00:55.575794 13816 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:55.577770 15343 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:55.581091 13608 tablet_copy_source_session.cc:205] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:55.590777 15397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ef275b7c0a45619cbc05266aea24dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:55.591055 15406 ts_tablet_manager.cc:927] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:55.598332 15406 tablet_copy_client.cc:316] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:00:55.601230 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet 232127da9f574f41a5a8087ae775ef6b from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:00:55.601836 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet 232127da9f574f41a5a8087ae775ef6b from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-232127da9f574f41a5a8087ae775ef6b
I20260807 08:00:55.614468 15404 ts_tablet_manager.cc:927] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:55.622622 13830 tablet_copy_source_session.cc:205] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:55.626883 15406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232127da9f574f41a5a8087ae775ef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:55.633714 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:55.648468 15404 tablet_copy_client.cc:316] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:55.650107 13608 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet 466eca488bfd44d396c22e2396d1ef7e from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:55.650683 13608 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet 466eca488bfd44d396c22e2396d1ef7e from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-466eca488bfd44d396c22e2396d1ef7e
I20260807 08:00:55.650223 15406 tablet_copy_client.cc:791] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:55.651520 15406 tablet_copy_client.cc:655] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:55.657385 15406 tablet_copy_client.cc:531] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:55.657980 15397 tablet_copy_client.cc:791] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:55.658526 15397 tablet_copy_client.cc:655] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:55.661120 13608 tablet_copy_source_session.cc:205] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:55.662029 15397 tablet_copy_client.cc:531] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:55.666532 15404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 466eca488bfd44d396c22e2396d1ef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:55.668560 15406 tablet_bootstrap.cc:492] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:55.684933 13816 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:55.686302 15364 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:55.705238 13705 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:55.707435 15364 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260807 08:00:55.723546 15406 tablet_bootstrap.cc:492] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:55.724397 15406 tablet_bootstrap.cc:492] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:55.725129 15406 ts_tablet_manager.cc:1397] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.057s user 0.021s sys 0.009s
I20260807 08:00:55.728322 15406 raft_consensus.cc:348] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.729935 15406 raft_consensus.cc:729] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:55.730753 15406 consensus_queue.cc:260] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.737890 15406 ts_tablet_manager.cc:1428] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260807 08:00:55.739400 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-232127da9f574f41a5a8087ae775ef6b received from {username='slave'} at 127.0.0.1:59332
I20260807 08:00:55.739792 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-232127da9f574f41a5a8087ae775ef6b on tablet 232127da9f574f41a5a8087ae775ef6b with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:55.760696 15397 tablet_bootstrap.cc:492] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:55.762754 15404 tablet_copy_client.cc:791] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:55.763475 15404 tablet_copy_client.cc:655] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:55.767452 15404 tablet_copy_client.cc:531] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:55.780619 15404 tablet_bootstrap.cc:492] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:55.792946 15397 tablet_bootstrap.cc:492] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:55.793901 15397 tablet_bootstrap.cc:492] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:55.794554 15397 ts_tablet_manager.cc:1397] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.010s
I20260807 08:00:55.797936 15397 raft_consensus.cc:348] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.799175 15397 raft_consensus.cc:729] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:55.799780 15397 consensus_queue.cc:260] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.802239 15397 ts_tablet_manager.cc:1428] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 08:00:55.805393 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-e5ef275b7c0a45619cbc05266aea24dc received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:55.805804 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-e5ef275b7c0a45619cbc05266aea24dc on tablet e5ef275b7c0a45619cbc05266aea24dc with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:55.836740 15404 tablet_bootstrap.cc:492] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:55.840121 15404 tablet_bootstrap.cc:492] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:55.840857 15404 ts_tablet_manager.cc:1397] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.010s
I20260807 08:00:55.843634 15404 raft_consensus.cc:348] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.844587 15404 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:55.845242 15404 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:55.847712 13592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:00:55.848265 13592 raft_consensus.cc:595] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:55.850867 13592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:00:55.851424 13592 raft_consensus.cc:595] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:55.851505 15404 ts_tablet_manager.cc:1428] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260807 08:00:55.852949 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-466eca488bfd44d396c22e2396d1ef7e received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:55.853322 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-466eca488bfd44d396c22e2396d1ef7e on tablet 466eca488bfd44d396c22e2396d1ef7e with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:55.855362 13584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:00:55.855835 13584 raft_consensus.cc:595] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:55.858539 15265 raft_consensus.cc:979] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:55.858956 15265 raft_consensus.cc:1067] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:55.860641 13816 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:00:55.860723 14629 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42236
I20260807 08:00:55.861343 13816 raft_consensus.cc:595] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:55.861399 14629 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:55.861934 14629 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.865481 13705 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:00:55.866058 13705 raft_consensus.cc:595] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:55.868636 13816 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:00:55.869215 13816 raft_consensus.cc:595] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:55.869072 14629 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:55.872258 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:55.872797 13807 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.874701 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260807 08:00:55.876351 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:55.876919 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:55.878696 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:55.879218 13592 raft_consensus.cc:3032] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:55.879500 13592 raft_consensus.cc:729] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:55.880751 13592 consensus_queue.cc:260] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:55.882112 13592 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.882282 13807 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:55.883397 14580 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:55.884231 15402 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:55.884691 15402 raft_consensus.cc:686] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:55.885499 15402 consensus_queue.cc:237] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:55.887621 13592 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:55.893256 14629 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:55.896348 13428 catalog_manager.cc:5495] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:55.943560 15374 raft_consensus.cc:979] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:55.944037 15374 raft_consensus.cc:1067] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:55.945833 13589 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56594
I20260807 08:00:55.946249 13589 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:55.946532 13589 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.957716 13589 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:55.960281 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:55.960906 13807 raft_consensus.cc:3032] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:55.961212 13807 raft_consensus.cc:729] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:55.960968 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:55.962003 13807 consensus_queue.cc:260] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:55.962884 13703 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.962992 13807 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:55.965508 13589 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:55.970866 13807 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:55.970866 13703 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:55.972064 13537 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:55.972803 15364 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:55.973269 15364 raft_consensus.cc:686] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:55.974139 15364 consensus_queue.cc:237] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:55.992132 14629 raft_consensus.cc:1203] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:55.997315 13428 catalog_manager.cc:5495] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:56.008020 14626 raft_consensus.cc:1203] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:56.104612 14626 raft_consensus.cc:1203] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:56.105571 15326 raft_consensus.cc:979] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:00:56.106089 15326 raft_consensus.cc:1067] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:00:56.107336 13807 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:54072
I20260807 08:00:56.107698 13807 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:56.108037 13807 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:56.114475 13807 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:56.120335 13807 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:56.122323 14626 raft_consensus.cc:1203] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:00:56.122570 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:56.123179 13703 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:56.123451 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:56.123844 13592 raft_consensus.cc:3032] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:00:56.124009 13592 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:56.124524 13592 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:56.125365 13592 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:56.128896 13703 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:00:56.135452 13769 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:56.133643 13592 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:00:56.136979 15433 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:00:56.137498 15433 raft_consensus.cc:686] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:56.138336 15433 consensus_queue.cc:237] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:56.150192 13428 catalog_manager.cc:5495] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 2 to 3, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:56.236615 15433 raft_consensus.cc:979] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:56.237135 15433 raft_consensus.cc:1067] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:56.238447 13592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56594
I20260807 08:00:56.238958 13592 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:56.239322 13592 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:56.245285 13592 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:56.246981 13592 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:56.247825 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:56.248402 13811 raft_consensus.cc:3032] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:56.248706 13811 raft_consensus.cc:729] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:56.249305 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:56.249866 13705 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:56.249452 13811 consensus_queue.cc:260] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:56.250562 13811 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:56.256379 13705 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:56.257498 13540 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:56.257820 13811 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:56.258646 15265 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:56.259318 15265 raft_consensus.cc:686] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:56.260043 15265 consensus_queue.cc:237] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:56.270221 13428 catalog_manager.cc:5495] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:56.286769 15258 raft_consensus.cc:979] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:56.287257 15432 raft_consensus.cc:1067] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:56.288666 14626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42252
I20260807 08:00:56.289132 14626 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:56.289435 14626 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:56.294507 14626 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.297063 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:56.297586 14626 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:56.297745 13705 raft_consensus.cc:3032] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:56.298058 13705 raft_consensus.cc:729] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:56.298696 13705 consensus_queue.cc:260] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.299765 13705 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:56.297138 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:56.312548 13811 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:56.316895 13705 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:56.318076 14583 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:56.318964 15343 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:56.319363 15343 raft_consensus.cc:686] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:56.320225 13811 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:56.320047 15343 consensus_queue.cc:237] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.328697 13428 catalog_manager.cc:5495] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:56.340967 13589 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:56.342175 15342 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:56.360932 15265 raft_consensus.cc:979] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:56.361361 15265 raft_consensus.cc:1067] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:56.365676 14626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42236
I20260807 08:00:56.366367 14626 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:56.366793 14626 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:56.375943 13703 raft_consensus.cc:1261] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:56.377624 15326 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:56.392016 14626 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.396065 13807 raft_consensus.cc:1261] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:56.397596 15265 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:56.398505 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:56.399111 13592 raft_consensus.cc:3032] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:56.399289 14626 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:56.400638 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:56.401208 13705 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:56.407900 13807 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:56.409399 15402 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:56.422739 13705 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:56.423795 14583 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:56.399417 13592 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:56.424674 15402 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:56.425186 15402 raft_consensus.cc:686] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:56.424820 13592 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.426064 15402 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.426571 13592 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:56.433820 13592 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:56.433929 15429 raft_consensus.cc:1050] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:56.436540 15429 consensus_queue.cc:237] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.472599 13428 catalog_manager.cc:5495] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:56.477497 14626 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.477627 13705 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.481076 13584 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.483114 15429 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.483965 15433 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.484921 15374 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.497048 15429 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.499614 14626 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.501955 13592 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.506424 15326 raft_consensus.cc:1050] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:56.508872 15326 consensus_queue.cc:237] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.508790 13705 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.526327 13432 catalog_manager.cc:5495] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.525447 13816 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.526356 14629 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.531289 15265 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.534641 15364 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.544114 13705 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.545288 15457 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:56.554610 13811 consensus_queue.cc:237] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.563956 14629 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:56.564531 13708 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:56.565084 15433 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:56.568590 15433 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:56.576253 15433 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.578701 13708 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.581995 14629 raft_consensus.cc:2932] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.588433 13703 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:56.589654 15246 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.596594 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet cc32413495ee41909c7c5494c8095084 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:56.603541 15457 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.611841 15433 raft_consensus.cc:1050] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:56.614367 15433 consensus_queue.cc:237] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.613111 13807 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.611622 13428 catalog_manager.cc:5495] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.632434 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet cc32413495ee41909c7c5494c8095084 with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:00:56.634753 15465 tablet_replica.cc:330] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:00:56.637960 15465 raft_consensus.cc:2229] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:56.638806 15465 raft_consensus.cc:2258] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:56.639012 13705 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.644331 13592 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:00:56.646064 15429 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.649688 14629 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.651587 13711 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.653307 15465 ts_tablet_manager.cc:1905] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:56.654186 13589 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.654086 15433 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:00:56.653016 14629 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.656028 15433 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.658519 15374 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.667517 13428 catalog_manager.cc:5495] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:56.679215 15433 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.684151 13584 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.696580 13591 consensus_queue.cc:237] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.703435 14626 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.705415 13816 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:56.710362 15465 ts_tablet_manager.cc:1918] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:56.710119 15326 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:00:56.711025 15465 log.cc:1196] T cc32413495ee41909c7c5494c8095084 P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/cc32413495ee41909c7c5494c8095084
I20260807 08:00:56.761288 13816 raft_consensus.cc:1261] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:56.763209 15265 consensus_queue.cc:1035] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:56.770859 13703 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:56.773947 14625 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:56.774178 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet cc32413495ee41909c7c5494c8095084 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:56.790382 15343 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:56.790683 15364 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:56.795689 13431 catalog_manager.cc:5495] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:56.798952 15364 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.799523 13711 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.800666 14625 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.815722 15457 raft_consensus.cc:1050] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:00:56.818558 15457 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.826313 13709 raft_consensus.cc:1261] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:56.830389 13811 consensus_queue.cc:237] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:56.837545 13589 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:56.838981 15402 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:56.832443 13807 raft_consensus.cc:2932] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.843828 15326 consensus_queue.cc:1035] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:56.847402 14626 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:56.851243 13709 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:56.856961 15343 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:56.857170 15317 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:56.897909 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e5ef275b7c0a45619cbc05266aea24dc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:56.906270 13807 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260807 08:00:56.891776 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 51462.4us
I20260807 08:00:56.916028 15481 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260807 08:00:56.927500 14629 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.934886 14626 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.938827 15481 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.939672 13592 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:00:56.942312 13705 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:00:56.944274 15481 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:56.944401 15433 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:56.962241 13589 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:56.978156 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:57.015862 13807 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:57.032537 15326 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:57.032660 15429 raft_consensus.cc:2932] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:57.178236 14630 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:57.191232 13431 catalog_manager.cc:5495] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:57.200227 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 232127da9f574f41a5a8087ae775ef6b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:00:57.221810 15343 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260807 08:00:57.237418 13706 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:57.278126 13591 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:00:57.278729 13591 raft_consensus.cc:595] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:57.294798 13591 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:00:57.295518 13591 raft_consensus.cc:595] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:00:57.328543 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 98377.6us
I20260807 08:00:57.343186 13809 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
W20260807 08:00:57.362905 15295 log.cc:924] Time spent T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260807 08:00:57.427552 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ef275b7c0a45619cbc05266aea24dc with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
W20260807 08:00:57.470583 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 244714us
W20260807 08:00:57.476567 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 134035us
W20260807 08:00:57.485579 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 140047us
I20260807 08:00:57.517057 15487 raft_consensus.cc:979] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:00:57.517597 15487 raft_consensus.cc:1067] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:00:57.537611 13807 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:54072
I20260807 08:00:57.538311 13807 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:57.538684 13807 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.585793 13807 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:57.602064 13807 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:00:57.611419 13706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "70356515c14242539aa745f7e028f6b6"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:00:57.617920 13706 raft_consensus.cc:595] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
W20260807 08:00:57.641707 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 79004.4us
I20260807 08:00:57.647388 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:57.648288 13589 raft_consensus.cc:3032] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:57.648783 13589 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:57.650682 13589 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:57.669036 13589 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.679533 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:00:57.680261 13811 raft_consensus.cc:595] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:57.683451 15524 tablet_replica.cc:330] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:57.685174 15524 raft_consensus.cc:2229] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:57.686347 15524 raft_consensus.cc:2258] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:57.702406 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:00:57.703285 13811 raft_consensus.cc:595] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:57.706097 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 08:00:57.707688 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:57.707736 13589 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:00:57.708312 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:57.724103 15521 raft_consensus.cc:979] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:57.724618 15521 raft_consensus.cc:1067] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:57.730901 13430 catalog_manager.cc:5495] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:57.748466 14626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42248
I20260807 08:00:57.749284 14626 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:57.749745 14626 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.835316 14626 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:57.848374 14626 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:57.848985 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:00:57.849781 14630 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.855620 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:57.856292 13710 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.875459 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:57.876096 13809 raft_consensus.cc:3032] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:57.876375 13809 raft_consensus.cc:729] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:57.879382 13809 consensus_queue.cc:260] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:57.880224 15524 ts_tablet_manager.cc:1905] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:57.880522 13809 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.895947 13809 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:57.896998 14580 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:57.902904 14630 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:00:57.914129 13769 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:57.914129 15485 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:57.923599 15521 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:57.906163 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:57.929765 15485 raft_consensus.cc:686] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:57.936304 13710 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:00:57.939930 13709 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.936978 15485 consensus_queue.cc:237] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:57.939059 15521 raft_consensus.cc:686] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:57.946352 15519 raft_consensus.cc:979] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:57.952208 15519 raft_consensus.cc:1067] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:57.949318 15522 raft_consensus.cc:979] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:57.953202 15528 raft_consensus.cc:1067] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:57.955668 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42236
I20260807 08:00:57.956281 14622 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:57.956624 14622 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.958372 14626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42248
I20260807 08:00:57.958882 14626 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:57.959192 14626 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.946380 15521 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:57.978636 14626 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:57.984618 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:57.985230 13591 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.989734 14626 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:57.991989 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:57.993202 13809 raft_consensus.cc:3032] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:57.993546 13809 raft_consensus.cc:729] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:57.994328 13809 consensus_queue.cc:260] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:57.995465 13809 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:57.998557 15371 raft_consensus.cc:979] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:58.001016 13709 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:58.002457 13591 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:58.006016 15371 raft_consensus.cc:1067] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:58.008473 14583 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:00:58.009799 15485 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:58.010418 15485 raft_consensus.cc:686] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:58.011557 15485 consensus_queue.cc:237] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:58.012696 13587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70356515c14242539aa745f7e028f6b6"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56592
I20260807 08:00:58.013300 13587 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:58.013677 13587 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:58.022519 15524 ts_tablet_manager.cc:1918] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:58.022286 14622 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:58.035827 13809 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:58.022938 15524 log.cc:1196] T e5ef275b7c0a45619cbc05266aea24dc P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/e5ef275b7c0a45619cbc05266aea24dc
I20260807 08:00:58.039482 14626 raft_consensus.cc:1226] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:00:58.042672 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet e5ef275b7c0a45619cbc05266aea24dc (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:00:58.046893 15364 consensus_queue.cc:1046] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260807 08:00:58.055768 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:58.057575 15488 raft_consensus.cc:3032] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:58.057956 15488 raft_consensus.cc:729] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:58.059612 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:58.060202 13709 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:58.052349 13587 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:58.058763 15488 consensus_queue.cc:260] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:58.068612 15488 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:58.069283 13709 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:58.070979 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:58.073334 15513 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:58.073853 15513 raft_consensus.cc:686] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:58.057014 14622 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:58.074672 15513 consensus_queue.cc:237] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:58.078384 13428 catalog_manager.cc:5495] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:58.088342 13587 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:58.093742 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:58.094312 13811 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:58.100757 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:58.101413 13709 raft_consensus.cc:3032] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:58.103001 13709 raft_consensus.cc:729] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:58.111253 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet 232127da9f574f41a5a8087ae775ef6b with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:00:58.112746 15524 tablet_replica.cc:330] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:00:58.113765 15524 raft_consensus.cc:2229] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:00:58.113832 13592 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:00:58.114418 15524 raft_consensus.cc:2258] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:00:58.119136 15524 ts_tablet_manager.cc:1905] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:00:58.121980 13709 consensus_queue.cc:260] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:58.125464 13709 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:58.123689 13432 catalog_manager.cc:5495] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:58.130823 13432 catalog_manager.cc:5495] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:58.132762 13432 catalog_manager.cc:5495] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:58.141691 13709 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:58.144814 13540 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:00:58.145597 13418 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 466eca488bfd44d396c22e2396d1ef7e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 77da774f2bb54cf28302d53d0db224ad is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:00:58.146526 15540 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:58.147138 15540 raft_consensus.cc:686] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:58.148170 13811 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:58.147931 15540 consensus_queue.cc:237] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:58.159420 13431 catalog_manager.cc:5495] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:58.180769 13809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:58.181293 13809 raft_consensus.cc:595] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Received request to transfer leadership
I20260807 08:00:58.187629 13809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:58.188068 13809 raft_consensus.cc:595] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Received request to transfer leadership
W20260807 08:00:58.188988 13415 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 466eca488bfd44d396c22e2396d1ef7e already in progress [suppressed 4 similar messages]
I20260807 08:00:58.192574 13809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54058
I20260807 08:00:58.193012 13809 raft_consensus.cc:595] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Received request to transfer leadership
W20260807 08:00:58.194197 13508 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 466eca488bfd44d396c22e2396d1ef7e already in progress
W20260807 08:00:58.194516 13508 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 466eca488bfd44d396c22e2396d1ef7e already in progress
I20260807 08:00:58.197300 13432 catalog_manager.cc:5495] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:58.200659 15524 ts_tablet_manager.cc:1918] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:00:58.201182 15524 log.cc:1196] T 232127da9f574f41a5a8087ae775ef6b P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/232127da9f574f41a5a8087ae775ef6b
I20260807 08:00:58.205900 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet 232127da9f574f41a5a8087ae775ef6b (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
W20260807 08:00:58.224263 13415 catalog_manager.cc:4584] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): ChangeConfig:REMOVE_PEER RPC failed for tablet 466eca488bfd44d396c22e2396d1ef7e: Remote error: Service unavailable: leader transfer in progress
I20260807 08:00:58.348475 13709 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:58.349818 15513 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:58.384981 13809 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:58.386654 15502 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:58.411336 13709 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 08:00:58.412309 13809 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:58.412590 15528 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:58.413650 15502 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:58.427129 15509 raft_consensus.cc:979] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:00:58.427601 15509 raft_consensus.cc:1067] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Signalling peer 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:00:58.431454 13584 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:58.432969 15485 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:58.435146 13709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39150
I20260807 08:00:58.435844 13709 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:58.436272 13709 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:58.442356 13709 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:58.444990 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:58.445163 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:58.445977 13592 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:58.446122 13811 raft_consensus.cc:3032] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:00:58.446213 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:00:58.446653 13811 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:58.446910 14626 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:58.447448 13811 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:58.451221 13811 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:00:58.445284 13709 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:00:58.454661 13592 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:00:58.457291 14626 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:00:58.460704 13709 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:58.461776 13811 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:00:58.462110 15485 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:58.464776 13657 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:58.465579 15491 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:00:58.465297 13587 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:58.466116 15491 raft_consensus.cc:686] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:58.466974 15491 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:58.467650 15502 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:00:58.511116 13431 catalog_manager.cc:5495] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 2 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 3 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:58.740954 13703 raft_consensus.cc:1261] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:58.742162 15572 consensus_queue.cc:1035] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:58.756659 13811 raft_consensus.cc:1261] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:58.760689 15572 consensus_queue.cc:1035] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:58.849480 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:58.887207 13709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:00:58.887669 13709 raft_consensus.cc:595] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:58.889667 13709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:00:58.890034 13709 raft_consensus.cc:595] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:58.892570 13709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:00:58.893095 13709 raft_consensus.cc:595] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:58.896353 13703 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:00:58.896860 13703 raft_consensus.cc:595] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:00:58.899677 13584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:00:58.900107 13584 raft_consensus.cc:595] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:00:58.901173 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 08:00:58.902835 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:00:58.903438 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:58.917672 13584 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:00:58.918754 15514 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:58.928879 14622 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:00:58.930313 15514 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:58.938642 15503 raft_consensus.cc:979] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:58.939060 15503 raft_consensus.cc:1067] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:00:58.943014 13811 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:00:58.944411 15258 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:58.948743 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42252
I20260807 08:00:58.949291 14622 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:58.952270 14622 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:58.973300 14622 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:58.974522 14630 raft_consensus.cc:1226] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:00:58.975858 14622 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:58.977604 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:58.978175 13584 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:58.978077 15503 consensus_queue.cc:1046] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:00:58.978726 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:58.979135 15503 raft_consensus.cc:3032] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:00:58.979477 15503 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:58.980240 15503 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:58.982359 15503 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:58.984417 15258 raft_consensus.cc:979] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:00:58.984660 13584 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:00:58.984903 15258 raft_consensus.cc:1067] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:00:58.986300 13809 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:54086
I20260807 08:00:58.986797 13809 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:58.987097 13809 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:58.997071 13703 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:00:59.000485 14583 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:59.001698 15513 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:00:59.001830 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:59.002291 15513 raft_consensus.cc:686] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:00:59.002480 13811 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:00:59.003175 15513 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:59.000057 13809 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:59.009212 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:59.009827 13584 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.010953 13811 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:00:59.009837 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:59.015483 13703 raft_consensus.cc:3032] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:59.015853 13703 raft_consensus.cc:729] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:59.013741 13431 catalog_manager.cc:5495] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 3 to 4, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:00:59.016805 13703 consensus_queue.cc:260] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:59.009898 13809 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:59.018217 13703 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.023682 13584 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:00:59.024127 13703 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:00:59.024854 13769 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:59.025681 15568 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:59.026132 15568 raft_consensus.cc:686] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:59.026881 15568 consensus_queue.cc:237] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:59.036140 13431 catalog_manager.cc:5495] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:59.076628 15514 raft_consensus.cc:979] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:59.077117 15514 raft_consensus.cc:1067] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:59.078756 13584 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56592
I20260807 08:00:59.079342 13584 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:59.079746 13584 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.085585 13584 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:59.087059 13584 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:00:59.087951 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:00:59.088038 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:59.088776 14626 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.088820 13703 raft_consensus.cc:3032] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:59.089397 13703 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:59.090241 13703 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:59.091732 13703 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.099064 13703 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:59.100052 13540 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:59.101408 15364 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:59.101909 15364 raft_consensus.cc:686] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:59.102689 15364 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:00:59.107834 14626 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:59.112270 15573 raft_consensus.cc:979] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:00:59.112695 15573 raft_consensus.cc:1067] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:00:59.114368 13809 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:54072
I20260807 08:00:59.114867 13809 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:59.115139 13809 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.119908 13809 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:59.121613 13809 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:59.122757 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:00:59.123085 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:59.123371 13587 raft_consensus.cc:3032] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:59.123632 15514 raft_consensus.cc:979] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:59.123723 13587 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:59.124102 15514 raft_consensus.cc:1067] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:00:59.124528 13587 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:59.125638 13584 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56592
I20260807 08:00:59.125764 13587 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.126158 13584 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:00:59.126514 13584 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.131531 13431 catalog_manager.cc:5495] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:59.132990 13587 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:00:59.123718 13703 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.133447 13584 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:59.134161 13769 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:00:59.135030 15570 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:59.135712 15570 raft_consensus.cc:686] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:00:59.136198 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:00:59.136757 13710 raft_consensus.cc:3032] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:00:59.137041 13710 raft_consensus.cc:729] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:00:59.136500 15570 consensus_queue.cc:237] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:59.137760 13710 consensus_queue.cc:260] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:59.138800 13710 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.145527 13703 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:00:59.151525 13432 catalog_manager.cc:5495] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:00:59.156244 13710 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:59.157150 13540 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:00:59.157265 13584 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:00:59.158923 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:00:59.159505 13809 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:00:59.159812 15364 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:00:59.160279 15364 raft_consensus.cc:686] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:00:59.161099 15364 consensus_queue.cc:237] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:00:59.167820 13809 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:00:59.180099 13432 catalog_manager.cc:5495] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:59.256321 13584 consensus_queue.cc:237] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.262892 13805 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260807 08:00:59.263023 13710 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260807 08:00:59.264168 15573 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:00:59.265023 15364 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260807 08:00:59.267483 13540 consensus_peers.cc:487] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c683c4293d64060bd1a28772f77225a. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:59.272267 15573 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.274039 13809 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.297375 13703 consensus_queue.cc:237] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.314790 13432 catalog_manager.cc:5495] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 3, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:59.335436 13584 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:59.335999 13703 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.336817 15510 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 08:00:59.338083 13661 consensus_peers.cc:487] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38e7d8fef17f4bdb92d0c21739339421. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:59.345333 15503 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.348588 13809 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:59.349862 15510 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.352281 13584 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
W20260807 08:00:59.357563 13418 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 466eca488bfd44d396c22e2396d1ef7e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260807 08:00:59.370994 13710 consensus_queue.cc:237] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.376694 13431 catalog_manager.cc:5495] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.384274 13809 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.389592 13811 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:59.390172 13661 consensus_peers.cc:487] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 966e99d2078c4b1c9fc921c1666b45a2. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:59.391189 15514 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.397579 13592 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:59.398813 15510 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.400527 15503 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.402642 13809 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.416301 13592 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.421638 13432 catalog_manager.cc:5495] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:00:59.422075 13710 consensus_queue.cc:237] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.438001 13811 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:00:59.438791 13661 consensus_peers.cc:487] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e110e8648fd04684ae2bf4d0c71752a3. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:00:59.439883 15258 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.439270 13592 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:00:59.442724 15258 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.460557 15258 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.462306 13587 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.463646 13811 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.479182 13431 catalog_manager.cc:5495] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:00:59.508574 13710 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:00:59.509097 13703 raft_consensus.cc:1261] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:59.509838 15485 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:00:59.510605 15583 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.521914 13587 raft_consensus.cc:1261] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:59.524809 13811 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:00:59.525373 15570 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:00:59.528122 13587 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:00:59.530297 15485 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:00:59.536150 15485 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:00:59.589228 14630 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:59.592060 15572 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:59.613636 13809 raft_consensus.cc:1261] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:59.615202 15572 consensus_queue.cc:1035] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.634711 13709 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:00:59.647336 15572 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:00:59.669104 13710 raft_consensus.cc:1261] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:59.675480 13584 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:59.680645 15364 consensus_queue.cc:1035] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.681437 15583 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.711293 13706 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:00:59.712704 15570 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:00:59.757266 15630 ts_tablet_manager.cc:927] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037)
I20260807 08:00:59.761421 15630 tablet_copy_client.cc:316] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.65:41037
I20260807 08:00:59.764330 13608 tablet_copy_service.cc:140] P 77da774f2bb54cf28302d53d0db224ad: Received BeginTabletCopySession request for tablet 5c683c4293d64060bd1a28772f77225a from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:33050)
I20260807 08:00:59.765065 13608 tablet_copy_service.cc:161] P 77da774f2bb54cf28302d53d0db224ad: Beginning new tablet copy session on tablet 5c683c4293d64060bd1a28772f77225a from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:33050: session id = 22ff0a1f4a4241d794ab182432c85893-5c683c4293d64060bd1a28772f77225a
I20260807 08:00:59.773866 13608 tablet_copy_source_session.cc:205] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:59.777164 15630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c683c4293d64060bd1a28772f77225a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:59.786377 15632 ts_tablet_manager.cc:927] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:59.788347 15632 tablet_copy_client.cc:316] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:00:59.791019 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet 966e99d2078c4b1c9fc921c1666b45a2 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:00:59.791595 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet 966e99d2078c4b1c9fc921c1666b45a2 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-966e99d2078c4b1c9fc921c1666b45a2
I20260807 08:00:59.793231 15630 tablet_copy_client.cc:791] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:59.793897 15630 tablet_copy_client.cc:655] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:59.797447 15630 tablet_copy_client.cc:531] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:59.799332 13724 tablet_copy_source_session.cc:205] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:59.802590 15632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966e99d2078c4b1c9fc921c1666b45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:59.806003 15630 tablet_bootstrap.cc:492] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:59.817488 15632 tablet_copy_client.cc:791] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:59.818037 15632 tablet_copy_client.cc:655] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:59.822364 15632 tablet_copy_client.cc:531] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:59.830431 15632 tablet_bootstrap.cc:492] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:59.836895 15630 tablet_bootstrap.cc:492] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:59.837915 15630 tablet_bootstrap.cc:492] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:59.838661 15630 ts_tablet_manager.cc:1397] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.005s
I20260807 08:00:59.841387 15630 raft_consensus.cc:348] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.842387 15630 raft_consensus.cc:729] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:59.843092 15630 consensus_queue.cc:260] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.847752 15636 ts_tablet_manager.cc:927] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:59.848795 15630 ts_tablet_manager.cc:1428] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260807 08:00:59.854751 13608 tablet_copy_service.cc:342] P 77da774f2bb54cf28302d53d0db224ad: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-5c683c4293d64060bd1a28772f77225a received from {username='slave'} at 127.0.0.1:33050
I20260807 08:00:59.855268 13608 tablet_copy_service.cc:433] P 77da774f2bb54cf28302d53d0db224ad: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-5c683c4293d64060bd1a28772f77225a on tablet 5c683c4293d64060bd1a28772f77225a with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:59.860332 15632 tablet_bootstrap.cc:492] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:59.861204 15632 tablet_bootstrap.cc:492] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:59.861904 15632 ts_tablet_manager.cc:1397] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.000s
I20260807 08:00:59.864459 15632 raft_consensus.cc:348] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.865339 15632 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:59.865909 15632 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.867970 15632 ts_tablet_manager.cc:1428] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 08:00:59.870242 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-966e99d2078c4b1c9fc921c1666b45a2 received from {username='slave'} at 127.0.0.1:45382
I20260807 08:00:59.870625 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-966e99d2078c4b1c9fc921c1666b45a2 on tablet 966e99d2078c4b1c9fc921c1666b45a2 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:59.878072 15636 tablet_copy_client.cc:316] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:00:59.879667 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet 38e7d8fef17f4bdb92d0c21739339421 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:00:59.880189 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet 38e7d8fef17f4bdb92d0c21739339421 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-38e7d8fef17f4bdb92d0c21739339421
I20260807 08:00:59.887499 13724 tablet_copy_source_session.cc:205] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:59.891005 15636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e7d8fef17f4bdb92d0c21739339421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:59.907157 15636 tablet_copy_client.cc:791] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:59.907692 15636 tablet_copy_client.cc:655] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:59.911509 15636 tablet_copy_client.cc:531] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:00:59.919955 15636 tablet_bootstrap.cc:492] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:00:59.924955 15632 ts_tablet_manager.cc:927] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:00:59.933319 15632 tablet_copy_client.cc:316] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:00:59.934914 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet e110e8648fd04684ae2bf4d0c71752a3 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:00:59.935443 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet e110e8648fd04684ae2bf4d0c71752a3 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-e110e8648fd04684ae2bf4d0c71752a3
I20260807 08:00:59.939982 15636 tablet_bootstrap.cc:492] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:00:59.940861 15636 tablet_bootstrap.cc:492] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:00:59.941577 15636 ts_tablet_manager.cc:1397] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.008s
I20260807 08:00:59.944604 15636 raft_consensus.cc:348] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.945542 15636 raft_consensus.cc:729] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:00:59.946750 13724 tablet_copy_source_session.cc:205] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:00:59.946213 15636 consensus_queue.cc:260] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:00:59.949241 15632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e110e8648fd04684ae2bf4d0c71752a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:00:59.952855 15636 ts_tablet_manager.cc:1428] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260807 08:00:59.954303 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-38e7d8fef17f4bdb92d0c21739339421 received from {username='slave'} at 127.0.0.1:45382
I20260807 08:00:59.954665 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-38e7d8fef17f4bdb92d0c21739339421 on tablet 38e7d8fef17f4bdb92d0c21739339421 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:00:59.951232 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:00:59.965276 15632 tablet_copy_client.cc:791] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:00:59.966150 15632 tablet_copy_client.cc:655] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:00:59.973356 15632 tablet_copy_client.cc:531] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:01:00.004549 15632 tablet_bootstrap.cc:492] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:01:00.036242 15632 tablet_bootstrap.cc:492] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:01:00.037164 15632 tablet_bootstrap.cc:492] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:01:00.037829 15632 ts_tablet_manager.cc:1397] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.004s
I20260807 08:01:00.038776 13584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1588644cd4f943a7b5d776326af48218"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:00.040346 13584 raft_consensus.cc:595] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:00.040531 15632 raft_consensus.cc:348] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:00.041467 15632 raft_consensus.cc:729] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:01:00.042127 15632 consensus_queue.cc:260] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:00.043498 13805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:00.044101 13805 raft_consensus.cc:595] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:00.044490 15632 ts_tablet_manager.cc:1428] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 08:01:00.045553 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260807 08:01:00.047179 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:00.047757 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:00.050412 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-e110e8648fd04684ae2bf4d0c71752a3 received from {username='slave'} at 127.0.0.1:45382
I20260807 08:01:00.050848 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-e110e8648fd04684ae2bf4d0c71752a3 on tablet e110e8648fd04684ae2bf4d0c71752a3 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:00.156045 15568 raft_consensus.cc:979] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:00.156601 15568 raft_consensus.cc:1067] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:00.159262 15573 raft_consensus.cc:979] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:00.159737 15573 raft_consensus.cc:1067] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:00.165400 13709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1588644cd4f943a7b5d776326af48218"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39134
I20260807 08:01:00.166043 13709 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:00.166433 13709 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:00.174611 13591 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56594
I20260807 08:01:00.175199 13591 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:00.175565 13591 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:00.179163 13709 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:00.182845 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:00.183395 13587 raft_consensus.cc:3032] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:00.183681 13587 raft_consensus.cc:729] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:00.184309 13587 consensus_queue.cc:260] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:00.184667 13709 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:00.185326 13587 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:00.188197 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:00.188746 13809 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:00.190009 13591 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:00.192574 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:00.193135 13805 raft_consensus.cc:3032] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:00.193186 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:00.193490 13805 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:00.193805 13710 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:00.194309 13805 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:00.195407 13805 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:00.203174 13805 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:00.208575 13587 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:00.209373 13591 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:00.208400 13710 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:00.210050 13661 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:00.211303 13540 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:00.213578 15626 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:00.213687 15503 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:00.215755 13809 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:00.218672 15503 raft_consensus.cc:686] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:00.219684 15503 consensus_queue.cc:237] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:00.222014 15626 raft_consensus.cc:686] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:00.222968 15626 consensus_queue.cc:237] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:00.247352 13431 catalog_manager.cc:5495] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:00.249749 14630 raft_consensus.cc:1203] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260807 08:01:00.284750 13428 catalog_manager.cc:5495] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 2 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:00.376850 14630 raft_consensus.cc:1203] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:01:00.390328 14630 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:00.394881 13709 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260807 08:01:00.395692 13591 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260807 08:01:00.396215 15646 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:00.400341 15646 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:01:00.404400 14630 raft_consensus.cc:1203] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:01:00.412245 15513 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 3 to 7, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:00.415019 13591 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 3 to 7, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:00.419457 13709 raft_consensus.cc:2932] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 3 to 7, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:00.433498 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 466eca488bfd44d396c22e2396d1ef7e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 8)
I20260807 08:01:00.433596 13428 catalog_manager.cc:5495] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 3 to 7, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:00.445829 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet 466eca488bfd44d396c22e2396d1ef7e with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54058
I20260807 08:01:00.447584 15657 tablet_replica.cc:330] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:01:00.448637 15657 raft_consensus.cc:2229] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260807 08:01:00.449576 15657 raft_consensus.cc:2258] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260807 08:01:00.453239 15657 ts_tablet_manager.cc:1905] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:01:00.470058 15657 ts_tablet_manager.cc:1918] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20260807 08:01:00.470434 15657 log.cc:1196] T 466eca488bfd44d396c22e2396d1ef7e P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/466eca488bfd44d396c22e2396d1ef7e
I20260807 08:01:00.472816 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet 466eca488bfd44d396c22e2396d1ef7e (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:01:00.477327 14630 raft_consensus.cc:1203] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:01:00.736547 13805 raft_consensus.cc:1261] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:00.737867 13809 raft_consensus.cc:1261] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:00.739346 15647 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:00.739487 15258 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:00.743621 13706 raft_consensus.cc:1261] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:00.744982 15626 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:00.802786 13591 raft_consensus.cc:1261] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:00.813097 15626 raft_consensus.cc:1050] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:01:00.853057 15503 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:00.862401 15626 consensus_queue.cc:237] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:00.884900 14626 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEARNER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 08:01:00.885697 13805 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 08:01:00.887440 13709 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 08:01:00.888419 15674 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:00.890275 15674 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:00.891166 15647 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:00.916749 15258 raft_consensus.cc:1050] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:01:00.918165 15674 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:00.919073 15258 consensus_queue.cc:237] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:00.921834 13710 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:00.926939 13805 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:00.931447 15503 raft_consensus.cc:1050] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:01:00.936159 14626 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:00.933928 15503 consensus_queue.cc:237] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:00.950965 13432 catalog_manager.cc:5495] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 3 to 4, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:00.963093 13809 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:00.964352 14630 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:00.965576 15258 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:00.966102 13592 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:00.966609 15510 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:00.968717 15510 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:00.988696 15510 raft_consensus.cc:1050] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:01:00.991235 15510 consensus_queue.cc:237] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:00.993901 15258 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.005239 13809 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.019059 14630 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:01.019455 14622 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:01.018498 14626 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.020448 13805 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:01.022862 15699 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260807 08:01:01.022814 15697 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260807 08:01:01.028915 13809 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:01.030310 15258 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:01.038700 13592 consensus_queue.cc:237] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.053561 13709 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:01.052498 13431 catalog_manager.cc:5495] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.061374 15647 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:01.072458 15572 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.080380 13710 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.086839 14626 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:01.089396 15626 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260807 08:01:01.093720 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 72075.6us
I20260807 08:01:01.097051 14622 raft_consensus.cc:2932] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.100277 13591 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:01.101622 13587 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.103166 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:01.098374 13592 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:01.096552 15701 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260807 08:01:01.124049 15258 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:01.124845 15701 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:01.131703 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5c683c4293d64060bd1a28772f77225a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:01:01.153795 13428 catalog_manager.cc:5495] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 4 to 5, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.168752 15697 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.172045 15258 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.173647 13706 consensus_queue.cc:237] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.173812 13591 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.185704 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c683c4293d64060bd1a28772f77225a with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54058
I20260807 08:01:01.187681 13587 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.197429 13811 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.198119 14630 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.225654 13710 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:01:01.226488 13710 raft_consensus.cc:595] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:01.232628 15713 tablet_replica.cc:330] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:01:01.234340 15713 raft_consensus.cc:2229] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260807 08:01:01.237558 13805 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.239282 15713 raft_consensus.cc:2258] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260807 08:01:01.226539 13432 catalog_manager.cc:5495] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } attrs { replace: true } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.240962 14629 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.245702 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:01.246279 13811 raft_consensus.cc:595] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:01.246347 13430 catalog_manager.cc:5495] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.249706 13587 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:01.250419 13584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:01.251048 13584 raft_consensus.cc:595] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:01.249818 14630 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:01.254884 15701 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:01.257050 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 08:01:01.258692 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:01.259577 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:01.275287 15713 ts_tablet_manager.cc:1905] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:01:01.277808 15699 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:01.279645 15699 raft_consensus.cc:979] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:01:01.280925 15510 raft_consensus.cc:1067] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:01:01.292733 15713 ts_tablet_manager.cc:1918] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260807 08:01:01.293335 15713 log.cc:1196] T 5c683c4293d64060bd1a28772f77225a P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/5c683c4293d64060bd1a28772f77225a
I20260807 08:01:01.295125 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet 5c683c4293d64060bd1a28772f77225a (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:01:01.301638 15510 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.306799 14629 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42252
I20260807 08:01:01.307382 14629 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:01.307704 14629 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:01.310295 13709 consensus_queue.cc:237] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.316941 13591 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.327318 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 966e99d2078c4b1c9fc921c1666b45a2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:01:01.335325 14626 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:01.335743 13587 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:01.338428 13706 consensus_queue.cc:237] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.340747 15572 raft_consensus.cc:979] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:01.339345 15699 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:01.341225 15572 raft_consensus.cc:1067] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:01.346451 15694 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:01.348400 13811 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:01.315441 14629 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.350136 14626 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:01.350522 15503 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:01.352587 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:01.353166 13587 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:01.358681 14629 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:01.357963 13428 catalog_manager.cc:5495] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:01.360234 13706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39134
I20260807 08:01:01.360810 13706 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:01.360082 15699 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.361177 13706 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:01.369719 13805 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.375028 15694 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:01.376490 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet 966e99d2078c4b1c9fc921c1666b45a2 with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
I20260807 08:01:01.377678 15713 tablet_replica.cc:330] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:01:01.377218 15510 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.378894 15713 raft_consensus.cc:2229] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:01:01.380959 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:01.381592 13710 raft_consensus.cc:3032] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:01.381880 13710 raft_consensus.cc:729] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:01.382588 13710 consensus_queue.cc:260] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.383762 13710 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:01.384948 13587 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:01.386395 14583 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:01.390406 13706 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.391289 15646 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:01.395222 15646 raft_consensus.cc:686] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:01.396104 15646 consensus_queue.cc:237] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.394691 13432 catalog_manager.cc:5495] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.400321 13706 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:01.401717 13710 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:01.401950 15713 raft_consensus.cc:2258] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:01:01.408782 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e7d8fef17f4bdb92d0c21739339421 with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:01:01.416549 15713 ts_tablet_manager.cc:1905] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:01:01.419890 15717 tablet_replica.cc:330] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:01:01.421011 15717 raft_consensus.cc:2229] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:01:01.421792 15717 raft_consensus.cc:2258] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:01:01.424086 14629 raft_consensus.cc:2932] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.427852 15717 ts_tablet_manager.cc:1905] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:01:01.424095 14625 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.428457 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:01.428861 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:01.429029 14626 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:01.424750 14630 raft_consensus.cc:2932] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.434053 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e110e8648fd04684ae2bf4d0c71752a3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:01:01.435689 14626 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:01.436522 13587 raft_consensus.cc:3032] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:01.436836 13587 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:01.437682 13587 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.438886 13587 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:01.447297 15717 ts_tablet_manager.cc:1918] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:01:01.447824 15717 log.cc:1196] T 38e7d8fef17f4bdb92d0c21739339421 P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/38e7d8fef17f4bdb92d0c21739339421
I20260807 08:01:01.449616 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet 38e7d8fef17f4bdb92d0c21739339421 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:01:01.453824 13587 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:01.459414 13661 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:01.460173 15503 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:01.462729 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 38e7d8fef17f4bdb92d0c21739339421 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:01:01.462077 13432 catalog_manager.cc:5495] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:01.463194 15503 raft_consensus.cc:686] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:01.464246 15503 consensus_queue.cc:237] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:01.467857 13587 raft_consensus.cc:2932] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.470889 15713 ts_tablet_manager.cc:1918] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:01:01.471463 15713 log.cc:1196] T 966e99d2078c4b1c9fc921c1666b45a2 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/966e99d2078c4b1c9fc921c1666b45a2
I20260807 08:01:01.473707 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet 966e99d2078c4b1c9fc921c1666b45a2 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:01:01.500015 13432 catalog_manager.cc:5495] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 3 to 4, VOTER 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:01.509928 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet e110e8648fd04684ae2bf4d0c71752a3 with delete_type TABLET_DATA_TOMBSTONED (TS 7a889001d6494fe5b7cbbe7da1dae302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54058
I20260807 08:01:01.511247 15713 tablet_replica.cc:330] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302: stopping tablet replica
I20260807 08:01:01.512209 15713 raft_consensus.cc:2229] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:01:01.511076 13430 catalog_manager.cc:5495] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 2 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:01.512787 15713 raft_consensus.cc:2258] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:01:01.517184 15713 ts_tablet_manager.cc:1905] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:01:01.541476 15713 ts_tablet_manager.cc:1918] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:01:01.541951 15713 log.cc:1196] T e110e8648fd04684ae2bf4d0c71752a3 P 7a889001d6494fe5b7cbbe7da1dae302: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-2-root/wals/e110e8648fd04684ae2bf4d0c71752a3
I20260807 08:01:01.543650 13415 catalog_manager.cc:4841] TS 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969): tablet e110e8648fd04684ae2bf4d0c71752a3 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:01:01.685637 15649 raft_consensus.cc:979] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:01.686403 15570 raft_consensus.cc:1067] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:01.688169 13587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56594
I20260807 08:01:01.688661 13587 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:01.689028 13587 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:01.693665 13587 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:01.695215 13587 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:01.696758 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:01.696288 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:01.697973 13709 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:01.698073 13805 raft_consensus.cc:3032] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:01.698444 13805 raft_consensus.cc:729] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:01.699599 13805 consensus_queue.cc:260] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:01.700884 13805 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:01.705610 13709 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:01.707214 13540 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:01.707980 15674 raft_consensus.cc:2781] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:01.708424 15674 raft_consensus.cc:686] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:01.709246 15674 consensus_queue.cc:237] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:01.712062 13805 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:01.721974 13430 catalog_manager.cc:5495] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:01.789247 13709 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:01.790530 15715 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:01.799886 13591 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:01.805626 15715 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:01.879384 14625 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:01.880450 15510 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:01.888338 13591 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:01.894325 15503 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:02.260452 13811 raft_consensus.cc:1261] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:02.261669 15572 consensus_queue.cc:1035] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:02.275993 13709 raft_consensus.cc:1261] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:02.277628 15572 consensus_queue.cc:1035] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:02.329731 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:02.445298 13587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:02.446224 13587 raft_consensus.cc:595] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:02.449651 13591 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "74ccb94230614cc48e846580ad82e994"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:02.450399 13591 raft_consensus.cc:595] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:02.453418 13709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:01:02.454100 13709 raft_consensus.cc:595] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:02.457789 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 08:01:02.462419 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:02.463357 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:02.527334 15647 raft_consensus.cc:979] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:02.528076 15647 raft_consensus.cc:1067] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:02.530516 13710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74ccb94230614cc48e846580ad82e994"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39134
I20260807 08:01:02.531082 13710 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:02.531558 13710 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:02.544279 13710 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:02.548271 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:02.549183 13587 raft_consensus.cc:3032] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:02.549593 13587 raft_consensus.cc:729] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:02.550593 13587 consensus_queue.cc:260] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:02.552120 13587 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:02.557354 13710 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:02.559831 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:02.560483 14626 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:02.563890 13587 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:02.565032 13661 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:02.565924 15503 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:02.566411 15503 raft_consensus.cc:686] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:02.567304 15503 consensus_queue.cc:237] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:02.568812 14626 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:02.579304 13428 catalog_manager.cc:5495] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 2 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:02.625687 15674 raft_consensus.cc:979] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:01:02.626287 15674 raft_consensus.cc:1067] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:01:02.630303 13811 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:54072
I20260807 08:01:02.630926 13811 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:02.631289 13811 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:02.640844 13811 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:02.644476 13811 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:02.646272 13805 consensus_queue.cc:237] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:02.647202 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:02.647830 13587 raft_consensus.cc:3032] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:02.648281 13587 raft_consensus.cc:729] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:02.649135 13587 consensus_queue.cc:260] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:02.647761 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:02.650360 13710 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:02.650664 13587 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:02.656201 13710 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:02.658334 13587 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:02.658949 13709 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:01:02.659976 13591 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:01:02.661614 13769 consensus_peers.cc:487] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06a735af95cd49e1942d669d2e5d91c6. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:01:02.662374 13769 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:02.663210 15649 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:02.663662 15649 raft_consensus.cc:686] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:02.664628 15649 consensus_queue.cc:237] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:02.666646 15736 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:02.667990 15568 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:02.677945 15699 raft_consensus.cc:979] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:01:02.678658 15699 raft_consensus.cc:1067] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:01:02.683911 13811 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:54086
I20260807 08:01:02.686884 13811 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:02.687311 13811 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:02.693638 15649 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:02.694998 13811 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:02.694787 13705 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:02.691892 13428 catalog_manager.cc:5495] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:02.696851 13811 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:02.699707 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:02.700260 13710 raft_consensus.cc:3032] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:02.700547 13710 raft_consensus.cc:729] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:02.701345 13710 consensus_queue.cc:260] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:02.702575 13710 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:02.711026 13709 consensus_queue.cc:237] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:02.717749 13710 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:02.719185 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:02.719836 14626 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:02.727159 14626 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:02.731410 13805 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:01:02.733932 13591 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:01:02.732903 15503 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260807 08:01:02.737900 13661 consensus_peers.cc:487] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ccaeb8c899348e7959597174d89ed01. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:01:02.738327 13769 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:02.738976 13587 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:02.740530 15649 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:02.741271 15649 raft_consensus.cc:686] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:02.740940 15699 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:02.744764 15649 consensus_queue.cc:237] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:02.746788 13428 catalog_manager.cc:5495] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:02.797971 15699 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:02.802232 13591 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:02.806866 13430 catalog_manager.cc:5495] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260807 08:01:02.886819 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 79384.6us
I20260807 08:01:02.922709 13811 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:02.927112 13430 catalog_manager.cc:5495] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:02.955541 13710 consensus_queue.cc:237] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:02.967296 13811 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 08:01:02.968365 13661 consensus_peers.cc:487] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0248ade936924e628c46b8de6252af1b. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:01:02.969277 13592 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:01:02.969303 15503 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:02.970625 15699 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:02.977468 15503 raft_consensus.cc:2932] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:02.979262 13811 raft_consensus.cc:2932] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:02.986838 13580 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:02.988375 15759 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:02.992089 13816 consensus_queue.cc:237] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:02.996347 13580 raft_consensus.cc:2932] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:03.002524 13428 catalog_manager.cc:5495] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
W20260807 08:01:03.010746 13769 consensus_peers.cc:487] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 -> Peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123): Couldn't send request to peer 22ff0a1f4a4241d794ab182432c85893. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b09437796639470caac9548c71d72fb2. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:01:03.010861 13710 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:01:03.013461 15736 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:03.027882 13580 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:01:03.030185 15568 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:03.034689 15749 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:03.042752 13709 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:03.073235 14626 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:03.076946 15759 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260807 08:01:03.092290 13432 catalog_manager.cc:5495] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:03.090241 13592 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } } }
I20260807 08:01:03.098502 13580 raft_consensus.cc:1261] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:03.105679 15749 consensus_queue.cc:1035] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:03.166745 13709 raft_consensus.cc:1261] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:03.168044 15568 consensus_queue.cc:1035] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:03.173928 15784 ts_tablet_manager.cc:927] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:03.207854 15784 tablet_copy_client.cc:316] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:01:03.212751 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet 1ccaeb8c899348e7959597174d89ed01 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:01:03.213333 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet 1ccaeb8c899348e7959597174d89ed01 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-1ccaeb8c899348e7959597174d89ed01
I20260807 08:01:03.223999 13724 tablet_copy_source_session.cc:205] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:01:03.227334 15784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccaeb8c899348e7959597174d89ed01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:01:03.245010 15784 tablet_copy_client.cc:791] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:01:03.245723 15784 tablet_copy_client.cc:655] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:01:03.250607 15784 tablet_copy_client.cc:531] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:01:03.259891 15784 tablet_bootstrap.cc:492] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:01:03.260748 15790 ts_tablet_manager.cc:927] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:03.269620 15790 tablet_copy_client.cc:316] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:01:03.271368 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet 06a735af95cd49e1942d669d2e5d91c6 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:01:03.271927 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet 06a735af95cd49e1942d669d2e5d91c6 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-06a735af95cd49e1942d669d2e5d91c6
I20260807 08:01:03.281739 13830 tablet_copy_source_session.cc:205] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:01:03.284334 15784 tablet_bootstrap.cc:492] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:01:03.285265 15784 tablet_bootstrap.cc:492] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:01:03.285962 15784 ts_tablet_manager.cc:1397] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.006s
I20260807 08:01:03.286154 15790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a735af95cd49e1942d669d2e5d91c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:01:03.289215 15784 raft_consensus.cc:348] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:03.290361 15784 raft_consensus.cc:729] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:01:03.291042 15784 consensus_queue.cc:260] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:03.294137 15784 ts_tablet_manager.cc:1428] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260807 08:01:03.299922 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-1ccaeb8c899348e7959597174d89ed01 received from {username='slave'} at 127.0.0.1:45382
I20260807 08:01:03.300324 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-1ccaeb8c899348e7959597174d89ed01 on tablet 1ccaeb8c899348e7959597174d89ed01 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:03.305101 14626 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:03.306473 15568 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:03.307503 15790 tablet_copy_client.cc:791] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:01:03.308416 15790 tablet_copy_client.cc:655] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:01:03.314316 15790 tablet_copy_client.cc:531] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:01:03.316546 13709 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:03.318295 15568 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:03.325359 15790 tablet_bootstrap.cc:492] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:01:03.359145 15790 tablet_bootstrap.cc:492] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:01:03.359965 15790 tablet_bootstrap.cc:492] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:01:03.360620 15790 ts_tablet_manager.cc:1397] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.004s
I20260807 08:01:03.362895 15790 raft_consensus.cc:348] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:03.363746 15790 raft_consensus.cc:729] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:01:03.364326 15790 consensus_queue.cc:260] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:03.366410 15790 ts_tablet_manager.cc:1428] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260807 08:01:03.368754 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-06a735af95cd49e1942d669d2e5d91c6 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:01:03.369153 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-06a735af95cd49e1942d669d2e5d91c6 on tablet 06a735af95cd49e1942d669d2e5d91c6 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:03.521698 15790 ts_tablet_manager.cc:927] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:03.523406 15790 tablet_copy_client.cc:316] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.66:44033
I20260807 08:01:03.524927 13724 tablet_copy_service.cc:140] P 9210dbc4cc9c45e5b26a463b705d362c: Received BeginTabletCopySession request for tablet 0248ade936924e628c46b8de6252af1b from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:45382)
I20260807 08:01:03.525363 13724 tablet_copy_service.cc:161] P 9210dbc4cc9c45e5b26a463b705d362c: Beginning new tablet copy session on tablet 0248ade936924e628c46b8de6252af1b from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:45382: session id = 22ff0a1f4a4241d794ab182432c85893-0248ade936924e628c46b8de6252af1b
I20260807 08:01:03.531993 13724 tablet_copy_source_session.cc:205] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:01:03.535568 15790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0248ade936924e628c46b8de6252af1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:01:03.535487 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:03.539837 15784 ts_tablet_manager.cc:927] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: Initiating tablet copy from peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:03.546231 15784 tablet_copy_client.cc:316] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Beginning tablet copy session from remote peer at address 127.11.140.67:40969
I20260807 08:01:03.547824 13830 tablet_copy_service.cc:140] P 7a889001d6494fe5b7cbbe7da1dae302: Received BeginTabletCopySession request for tablet b09437796639470caac9548c71d72fb2 from peer 22ff0a1f4a4241d794ab182432c85893 ({username='slave'} at 127.0.0.1:59332)
I20260807 08:01:03.548322 13830 tablet_copy_service.cc:161] P 7a889001d6494fe5b7cbbe7da1dae302: Beginning new tablet copy session on tablet b09437796639470caac9548c71d72fb2 from peer 22ff0a1f4a4241d794ab182432c85893 at {username='slave'} at 127.0.0.1:59332: session id = 22ff0a1f4a4241d794ab182432c85893-b09437796639470caac9548c71d72fb2
I20260807 08:01:03.548764 15790 tablet_copy_client.cc:791] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:01:03.549480 15790 tablet_copy_client.cc:655] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:01:03.555771 13830 tablet_copy_source_session.cc:205] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:01:03.556643 15790 tablet_copy_client.cc:531] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:01:03.558893 15784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09437796639470caac9548c71d72fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:01:03.564453 15790 tablet_bootstrap.cc:492] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:01:03.584319 15784 tablet_copy_client.cc:791] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 0 data blocks...
I20260807 08:01:03.584945 15784 tablet_copy_client.cc:655] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Starting download of 1 WAL segments...
I20260807 08:01:03.592577 15784 tablet_copy_client.cc:531] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:01:03.609841 15784 tablet_bootstrap.cc:492] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap starting.
I20260807 08:01:03.610268 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:03.610874 13811 raft_consensus.cc:595] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:03.613587 13710 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:01:03.614167 13710 raft_consensus.cc:595] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:03.616854 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:03.617381 13811 raft_consensus.cc:595] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:03.618453 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 08:01:03.619988 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:03.620476 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:03.621451 15790 tablet_bootstrap.cc:492] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:01:03.622303 15790 tablet_bootstrap.cc:492] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:01:03.623067 15790 ts_tablet_manager.cc:1397] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.059s user 0.022s sys 0.005s
I20260807 08:01:03.625654 15790 raft_consensus.cc:348] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:03.626526 15790 raft_consensus.cc:729] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:01:03.627157 15790 consensus_queue.cc:260] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:03.629469 15790 ts_tablet_manager.cc:1428] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 08:01:03.633154 13724 tablet_copy_service.cc:342] P 9210dbc4cc9c45e5b26a463b705d362c: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-0248ade936924e628c46b8de6252af1b received from {username='slave'} at 127.0.0.1:45382
I20260807 08:01:03.633584 13724 tablet_copy_service.cc:433] P 9210dbc4cc9c45e5b26a463b705d362c: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-0248ade936924e628c46b8de6252af1b on tablet 0248ade936924e628c46b8de6252af1b with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:03.633870 15784 tablet_bootstrap.cc:492] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:01:03.634742 15784 tablet_bootstrap.cc:492] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: Bootstrap complete.
I20260807 08:01:03.636762 15784 ts_tablet_manager.cc:1397] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.000s
I20260807 08:01:03.639041 15784 raft_consensus.cc:348] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:03.639660 15784 raft_consensus.cc:729] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Initialized, Role: LEARNER
I20260807 08:01:03.640223 15784 consensus_queue.cc:260] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:03.643537 15784 ts_tablet_manager.cc:1428] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260807 08:01:03.646420 13830 tablet_copy_service.cc:342] P 7a889001d6494fe5b7cbbe7da1dae302: Request end of tablet copy session 22ff0a1f4a4241d794ab182432c85893-b09437796639470caac9548c71d72fb2 received from {username='slave'} at 127.0.0.1:59332
I20260807 08:01:03.646776 13830 tablet_copy_service.cc:433] P 7a889001d6494fe5b7cbbe7da1dae302: ending tablet copy session 22ff0a1f4a4241d794ab182432c85893-b09437796639470caac9548c71d72fb2 on tablet b09437796639470caac9548c71d72fb2 with peer 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:03.702787 14626 raft_consensus.cc:1203] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:01:03.706224 14626 raft_consensus.cc:1203] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:01:03.798632 15736 raft_consensus.cc:979] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:03.799113 15568 raft_consensus.cc:1067] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:03.800813 13580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56594
I20260807 08:01:03.801292 13580 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:03.801715 13580 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:03.810920 13580 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:03.812927 13580 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:03.813971 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:03.814667 13811 raft_consensus.cc:3032] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:03.814958 13811 raft_consensus.cc:729] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:03.815788 13811 consensus_queue.cc:260] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:03.816534 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:03.817062 13709 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:03.817091 13811 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:03.823182 13811 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:03.823220 13709 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:03.824632 13537 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:03.829813 15758 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:03.830987 15758 raft_consensus.cc:686] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:03.831894 15758 consensus_queue.cc:237] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:03.845132 13430 catalog_manager.cc:5495] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:03.936282 15503 raft_consensus.cc:979] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:01:03.936879 15759 raft_consensus.cc:1067] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:01:03.938773 13811 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:54086
I20260807 08:01:03.939320 13811 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:03.939649 13811 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:03.947106 13811 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:03.948625 13811 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:03.950069 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:03.951768 13580 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:03.950917 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:03.954389 14626 raft_consensus.cc:1203] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:01:03.954563 13709 raft_consensus.cc:3032] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:03.954869 13709 raft_consensus.cc:729] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:03.955590 13709 consensus_queue.cc:260] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:03.956614 13709 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:03.966457 13580 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:03.967574 13769 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:03.968336 15736 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:03.968822 15736 raft_consensus.cc:686] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:03.969882 13709 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:03.969673 15736 consensus_queue.cc:237] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:03.983842 13430 catalog_manager.cc:5495] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:04.026553 14626 raft_consensus.cc:1203] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260807 08:01:04.030164 15736 raft_consensus.cc:979] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:04.030611 15568 raft_consensus.cc:1067] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:04.031976 13580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56594
I20260807 08:01:04.032533 13580 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:04.032933 13580 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:04.040731 13580 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:04.043340 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:04.043437 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:04.043926 13580 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:04.044054 13811 raft_consensus.cc:3032] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:04.044480 13811 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:04.045172 13811 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:04.046257 13811 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:04.044054 13710 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:04.051692 13811 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:04.052695 13537 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:04.053443 15758 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:04.053938 15758 raft_consensus.cc:686] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:04.054790 15758 consensus_queue.cc:237] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:04.058029 13710 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:04.070699 13430 catalog_manager.cc:5495] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:04.093497 15736 raft_consensus.cc:1050] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:01:04.095868 15736 consensus_queue.cc:237] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:04.113245 14626 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:04.114025 13587 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:04.114095 13710 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:04.115679 15649 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:04.117918 15736 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260807 08:01:04.121583 15749 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:04.132133 15736 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.133800 14626 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.140223 15759 raft_consensus.cc:1050] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:01:04.142589 15759 consensus_queue.cc:237] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:04.150893 13709 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.153846 14626 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:04.154789 13587 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:04.157881 15503 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:04.159304 15503 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:04.165547 13585 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.170827 13816 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:04.172647 13430 catalog_manager.cc:5495] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.179476 15827 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:04.199477 15827 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.202493 13805 consensus_queue.cc:237] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:04.205749 14626 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.212037 13811 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.221138 14629 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:04.225459 15749 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:04.230937 13580 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.238580 13428 catalog_manager.cc:5495] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:04.247008 15649 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.259063 13415 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 06a735af95cd49e1942d669d2e5d91c6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:01:04.260012 13585 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:04.262202 14626 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.262233 15736 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:04.267189 13585 raft_consensus.cc:2932] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.278313 13431 catalog_manager.cc:5495] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:04.280222 13710 consensus_queue.cc:237] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:04.292251 13811 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:04.292232 14626 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260807 08:01:04.293975 15824 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:04.298080 15824 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:04.300899 15827 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.302232 13811 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.314980 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet 06a735af95cd49e1942d669d2e5d91c6 with delete_type TABLET_DATA_TOMBSTONED (TS 9210dbc4cc9c45e5b26a463b705d362c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39124
I20260807 08:01:04.317619 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1ccaeb8c899348e7959597174d89ed01 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260807 08:01:04.317853 15841 tablet_replica.cc:330] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c: stopping tablet replica
I20260807 08:01:04.327097 15841 raft_consensus.cc:2229] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:01:04.322350 13432 catalog_manager.cc:5495] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.327889 15841 raft_consensus.cc:2258] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:01:04.333388 15841 ts_tablet_manager.cc:1905] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:01:04.352455 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ccaeb8c899348e7959597174d89ed01 with delete_type TABLET_DATA_TOMBSTONED (TS 77da774f2bb54cf28302d53d0db224ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56576
I20260807 08:01:04.356904 15841 ts_tablet_manager.cc:1918] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:01:04.357347 15841 log.cc:1196] T 06a735af95cd49e1942d669d2e5d91c6 P 9210dbc4cc9c45e5b26a463b705d362c: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-1-root/wals/06a735af95cd49e1942d669d2e5d91c6
I20260807 08:01:04.358990 13418 catalog_manager.cc:4841] TS 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): tablet 06a735af95cd49e1942d669d2e5d91c6 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:01:04.358767 14626 raft_consensus.cc:2932] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:04.373183 13709 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:04.373723 15842 tablet_replica.cc:330] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad: stopping tablet replica
I20260807 08:01:04.375808 15758 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:04.377848 15842 raft_consensus.cc:2229] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:01:04.378619 15842 raft_consensus.cc:2258] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:01:04.382031 15842 ts_tablet_manager.cc:1905] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260807 08:01:04.406540 15842 ts_tablet_manager.cc:1918] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260807 08:01:04.407100 15842 log.cc:1196] T 1ccaeb8c899348e7959597174d89ed01 P 77da774f2bb54cf28302d53d0db224ad: Deleting WAL directory at /tmp/dist-test-taskCXRxZ7/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786089561396890-11825-0/minicluster-data/ts-0-root/wals/1ccaeb8c899348e7959597174d89ed01
I20260807 08:01:04.412187 13811 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:04.413225 13418 catalog_manager.cc:4841] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): tablet 1ccaeb8c899348e7959597174d89ed01 (table test-workload [id=caa87c2ca7174fbd91b7d98f264e2faa]) successfully deleted
I20260807 08:01:04.413403 15846 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:05.434208 15855 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:05.434978 15855 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.438333 15855 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:05.443130 15863 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.444000 15863 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.447820 15863 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:05.475386 15873 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.476508 15867 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:05.479641 15867 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.481848 15867 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:05.480648 15873 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:05.486133 15868 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.486703 15868 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:05.491724 15879 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:05.492213 15879 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.496181 15879 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:05.503212 15867 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.504207 15868 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:05.735246 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:05.517638 15873 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:05.506877 15867 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:05.845312 15863 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:05.845969 15863 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.847071 15867 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:05.855590 15859 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:05.855980 15863 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:05.856177 15859 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.856727 15854 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.857209 15854 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.858322 15859 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:05.502802 15877 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:05.859239 15854 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:05.854684 15858 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.862870 15876 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:05.862906 15858 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:05.863484 15876 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.865090 15858 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:05.861727 15855 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.859591 15866 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.854950 15867 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:05.860134 15871 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:05.860370 15863 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.865798 15858 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:05.865468 15876 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:05.867146 15859 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.877238 15854 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:05.866016 15855 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.927732 15867 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:05.928661 15871 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.868419 15877 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.934083 15866 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:05.903596 15863 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:05.962901 15854 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:05.950858 15859 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:05.955752 15893 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:05.946733 15876 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:05.950065 15858 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.005944 15895 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.123739 15884 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.136899 15893 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.124308 15873 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.144726 15876 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.164872 15876 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.165485 15876 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.166101 15876 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.168426 15876 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.190023 15867 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.190972 15866 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.191968 15877 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.192039 15863 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.191815 15873 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.193063 15859 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.193344 15854 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.149117 15855 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.144248 15884 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.144636 15895 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.196065 15896 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.195549 15876 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.199415 15873 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.197537 15894 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.197557 15867 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.223436 15896 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.223289 15913 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.199275 15866 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.199828 15863 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.200172 15854 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.223982 15914 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.252247 15902 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.257074 15873 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.256502 15918 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.253360 15855 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.232024 15871 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.265677 15896 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.226882 15867 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.225207 15876 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.272428 15858 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.270430 15899 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.231992 15884 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.274740 15907 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.277676 15891 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:06.276422 15904 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.280450 15877 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.278849 15893 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.314164 15889 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.314414 15901 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.273131 15854 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.287912 15896 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.307382 15858 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.282609 15911 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.289727 15885 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.273909 15863 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.306967 15881 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.276467 15859 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.281968 15855 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.277364 15866 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.288491 15918 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.307742 15873 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.229679 15894 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.271624 15902 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.270035 15914 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.233879 15913 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.326360 15891 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.317627 15904 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.317174 15877 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.332633 15896 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.307633 15899 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.375936 15867 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.402050 15915 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.333155 15858 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.407584 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:06.377413 15859 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.387231 15900 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.400653 15895 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.313591 15907 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.344336 15889 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.384277 15911 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.333887 15901 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.380146 15881 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.379614 15885 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.420435 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:06.417141 15870 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.616998 14626 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:01:06.610523 15893 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:06.618434 15929 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.505895 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:06.655719 13705 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:06.627033 15863 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.656623 15904 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.658406 15904 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.659170 15904 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.663444 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:06.520215 14664 debug-util.cc:398] Leaking SignalData structure 0x7b080026fd60 after lost signal to thread 11828
I20260807 08:01:06.617887 15859 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.442119 15884 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.464339 15914 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.618954 15922 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.522089 15926 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.467491 15876 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.444804 15913 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.484928 15944 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.602106 15867 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.670135 15879 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.672243 13705 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
W20260807 08:01:06.671058 14664 debug-util.cc:398] Leaking SignalData structure 0x7b080026f880 after lost signal to thread 13481
I20260807 08:01:06.507164 15894 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.676820 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:06.680227 15937 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.471449 15938 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.652856 15866 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.655159 15873 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.521099 15939 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:06.520665 15933 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.446169 15920 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.520040 15952 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.521644 15955 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.492457 15921 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.645546 15958 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.654433 15893 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:06.687412 14664 debug-util.cc:398] Leaking SignalData structure 0x7b08002ffbe0 after lost signal to thread 13630
I20260807 08:01:06.687378 13709 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
W20260807 08:01:06.688345 13657 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.179s user 0.126s sys 0.261s
I20260807 08:01:06.676187 15855 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.682761 15877 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.692847 15956 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.680138 15947 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.695500 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:06.677829 15884 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:06.686959 15858 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.697324 15963 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.689904 15902 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.690217 15931 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.418732 15915 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:06.703517 13657 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.194s user 0.126s sys 0.261s
I20260807 08:01:06.705670 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:06.706369 14664 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff380 after lost signal to thread 13743
I20260807 08:01:06.695341 15873 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.689436 15867 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.446121 15900 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.707310 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:06.691828 15879 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.670435 15929 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:06.720824 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 1.21181e+06us
I20260807 08:01:06.619091 15870 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:06.722070 14664 debug-util.cc:398] Leaking SignalData structure 0x7b080034f1a0 after lost signal to thread 13854
I20260807 08:01:06.711823 15899 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.726383 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:06.727723 14664 debug-util.cc:398] Leaking SignalData structure 0x7b08002dfa40 after lost signal to thread 14669
I20260807 08:01:06.704442 15938 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.728612 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:06.729030 15975 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.712241 15921 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.729298 13811 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:06.729710 15975 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.712430 15958 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.707693 15920 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.733568 15965 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.733968 15901 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.734546 15901 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.735152 15901 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.735605 15977 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.736127 13585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:06.736770 13585 raft_consensus.cc:595] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:06.737269 15901 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.738129 13705 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:06.692601 15922 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:06.688211 15944 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.689610 15926 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.743891 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:06.698782 15937 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.748310 15957 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.748418 15935 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.748611 15907 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.702744 15939 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.747113 13811 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:01:06.752363 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:06.738523 15940 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.711440 15952 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.738883 15899 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.748806 15877 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.741617 15855 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.754052 13805 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:01:06.746524 15904 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.758407 13816 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:06.732681 15873 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.736263 15977 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.763128 13816 raft_consensus.cc:595] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:06.754182 15913 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.704289 15933 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.734216 15965 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.756500 15907 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.761898 15911 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.776867 15881 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.721977 15947 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.756476 15954 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.757814 15897 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.781448 13703 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:06.782779 15977 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.742834 15964 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.776292 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:06.778945 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:06.766553 15904 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.788151 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:06.731422 15963 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.713454 15955 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.738222 15931 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.788461 15884 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.762598 15877 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.783429 15889 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.762531 15970 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.764643 15855 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.799576 13809 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:06.722936 15956 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.757596 15899 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.789175 15911 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.816062 15978 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.792428 15867 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.789800 15893 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.819108 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:06.792253 15881 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.780978 15913 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.765965 15935 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.766340 15957 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.827481 15937 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.827589 15944 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.828751 13587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:06.827771 15929 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.833423 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:06.793160 15907 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.816730 15879 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.810266 15977 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.836851 13587 raft_consensus.cc:595] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:06.805817 15904 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.797601 15973 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.791147 15972 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.839826 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:06.843786 15975 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.834502 15938 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.784571 15940 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.856253 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:06.841770 15965 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.836247 15911 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.850700 15889 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:06.814445 15964 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.836938 15881 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:06.802887 15897 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.850421 15976 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:06.814620 15954 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.850510 15977 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.831722 15970 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.854590 15973 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:06.842638 15978 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:06.922521 14583 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:06.870007 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:06.880772 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 08:01:06.887687 15952 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.853109 15915 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.114988 13711 raft_consensus.cc:1261] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:06.854849 15900 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:07.175376 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 359987us
I20260807 08:01:07.183858 14583 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.182730 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:07.184695 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.185053 15895 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.185497 15895 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:07.185925 15895 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.186093 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.189275 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.184849 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.190457 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.192209 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.192459 13805 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:07.195066 15911 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.936034 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.209492 15964 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.881003 15958 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.880519 15921 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.948374 15939 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.902591 15956 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.902971 15975 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.045436 15855 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.191372 15871 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.212010 15914 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:06.859678 15972 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.212956 15854 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:07.191185 15955 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.228888 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.209789 15970 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.213047 13589 raft_consensus.cc:1261] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:06.868448 15935 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.179535 15963 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.971643 15913 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.217149 15969 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.942122 15981 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:06.857728 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.036597 15868 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:06.971819 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.253162 14618 raft_consensus.cc:2454] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:01:07.115993 15904 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.045066 15957 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.254227 15978 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.262518 15927 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:07.263094 15927 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:07.253419 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.270327 15897 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.271608 15927 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:07.273227 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.273500 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.275933 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.276973 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.277501 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.277845 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.278031 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.278574 13709 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:07.278750 15895 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.279291 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.280001 15896 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:07.280794 15895 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.281108 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.281358 15885 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.282408 15870 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.283041 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.284065 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.043392 15893 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:06.922695 15873 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:06.937408 15926 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:06.942338 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:06.981453 15922 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:06.956878 15920 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.225551 15914 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:07.224285 15871 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:07.227039 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.316713 15871 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.227659 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.318368 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.191089 15907 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.195217 15881 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.232255 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.318975 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.229506 15972 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.322016 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:06.971596 15947 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:07.322311 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.323069 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.323426 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.324175 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.202579 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:07.234299 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.236956 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.043695 15931 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.238113 15940 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.238164 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.327965 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.238909 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.180609 15889 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:07.240383 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.329074 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.115355 15891 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.318113 15914 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.044415 15933 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:07.166320 15918 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:07.298475 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.329988 15889 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.330124 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.236330 15899 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.241133 15877 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.330703 13711 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:07.223310 15975 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.310056 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.310720 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.310817 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.218257 15976 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.315987 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.316319 15973 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.319715 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.338481 13591 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:07.321002 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.321115 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.325127 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.339581 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.325167 13585 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:07.328054 13814 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:07.239195 15984 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:07.332464 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.289052 15868 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.339383 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.336380 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.301878 15969 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.341965 13592 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:07.342022 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:07.334277 13661 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.010s sys 0.014s
W20260807 08:01:07.343400 13661 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.525s user 0.011s sys 0.014s
I20260807 08:01:07.309983 15893 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.343983 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.345392 13817 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:07.345626 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.345536 15984 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:07.352610 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.333537 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.353849 13590 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:01:07.354292 15984 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.344657 13580 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:07.345981 15868 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.326359 13702 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:07.299520 15981 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.359663 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.346029 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.333315 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.348033 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.348258 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.336247 13700 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:07.350695 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.343539 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:07.344089 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 525639us
I20260807 08:01:07.341279 15954 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.358884 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.344789 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.358980 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.359283 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.363744 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.357184 15977 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.372602 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.379048 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.379335 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.379061 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.373821 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.372629 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.358989 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.374961 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.375821 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.380767 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.382380 15879 raft_consensus.cc:1050] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:01:07.382548 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.382730 15981 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.380184 13580 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:07.383715 15871 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.384287 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.393009 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.435093 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.436015 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.440267 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.423167 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.386018 15969 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.387472 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.388059 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.436429 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.388849 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.389024 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.389763 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.392270 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.442174 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.385855 15893 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.454726 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.455257 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.435766 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.458107 13710 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:07.409130 13591 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:07.381443 13661 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.459725 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.460166 15984 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.460460 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.460599 15984 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:07.461040 15984 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.460162 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.461931 13808 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:07.461961 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.463472 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.463721 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.465198 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.465886 13816 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:07.465833 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.467387 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.467653 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.468336 13816 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:07.468420 15984 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.468791 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:07.469607 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 88250.3us
I20260807 08:01:07.470352 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.470439 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.470693 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.471900 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.474472 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.473899 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.477337 13540 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.477903 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.478062 15969 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.478480 15969 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:07.478830 15969 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.479664 15984 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.479821 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.484715 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.484830 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.486229 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.486239 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.487879 13769 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:07.488692 15933 raft_consensus.cc:2781] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.489068 15933 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:07.489429 15933 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.494814 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.495491 15975 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.495890 15954 raft_consensus.cc:979] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:07.495754 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.496325 15954 raft_consensus.cc:1067] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Signalling peer 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:07.496552 15871 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.490345 15922 raft_consensus.cc:1050] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: attempting to promote NON_VOTER 22ff0a1f4a4241d794ab182432c85893 to VOTER
I20260807 08:01:07.490283 15879 consensus_queue.cc:237] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.500294 15922 consensus_queue.cc:237] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.502274 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.502341 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.503204 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.507818 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.507978 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.509132 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.510010 14583 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.511251 14583 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.511842 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.512058 15895 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.512428 15895 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:07.512763 15895 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.513322 15976 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.514706 15871 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.515106 15871 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:07.515442 15871 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.518615 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.519505 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.520284 13817 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:07.521404 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.522433 15895 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.523368 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.525336 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.498149 15977 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260807 08:01:07.428694 13588 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:07.527653 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.529500 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.530314 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.531383 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.531968 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.532152 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.528656 15895 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.534322 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.535621 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.535651 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.535967 14583 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.537079 15895 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.537494 15895 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:07.537963 15895 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.541934 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.543577 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.545540 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.548102 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.547876 15933 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.553819 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.553871 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.555212 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.555660 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.555588 15969 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:07.555126 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:07.556427 15914 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.558038 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.558410 14626 raft_consensus.cc:1226] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:07.558954 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.559846 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.561630 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.562086 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:07.563107 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:07.563742 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:07.565140 13657 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.566689 15914 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:07.567384 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.565188 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.569552 15895 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.572417 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.573410 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.576844 13657 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.578511 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.578482 15984 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.579733 15984 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.580199 15984 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:07.587903 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 75133.3us
I20260807 08:01:07.588603 15969 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.589731 15986 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.589359 15871 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.590114 15986 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:07.589984 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.590667 15986 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.592098 13657 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.592235 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.593376 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.596515 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:07.597440 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.598924 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:07.591602 15871 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.600101 15940 raft_consensus.cc:979] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:01:07.601025 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.602396 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.602568 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:07.603536 15940 raft_consensus.cc:1067] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:01:07.603964 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.601210 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.605444 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.606607 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.594969 15977 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.595191 15868 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.608620 15868 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:07.609011 15868 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.613781 13540 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.614743 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.616617 13540 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.617671 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.618760 13540 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.619561 15931 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.619853 15933 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.612088 15969 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.626109 15920 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:07.629976 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.615899 15977 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.631166 15975 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:07.631762 15975 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.634174 15954 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.630553 15895 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.636358 13537 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.637451 15947 raft_consensus.cc:979] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:07.637909 15947 raft_consensus.cc:1067] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:07.638765 15868 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:07.639897 14583 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.641245 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.642941 13540 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.644131 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.644901 15899 raft_consensus.cc:2726] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.645160 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.645540 15909 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:07.646238 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.643791 14583 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.647292 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.648196 15921 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.648576 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.648530 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.642060 13661 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:07.650592 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.651485 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.652216 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.652527 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.653497 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.653784 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.654177 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.654544 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.653242 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.654929 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.656872 13657 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:07.657478 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.658113 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.648190 15981 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.658679 13816 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:07.658800 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.659097 15891 raft_consensus.cc:2726] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.659438 14580 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.659734 15891 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.658700 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:07.660182 15891 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:07.660569 15891 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.660625 15899 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.660751 13657 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.661085 15899 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:07.661511 15899 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.662179 13537 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.664218 13769 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.664376 15926 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.663244 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.666426 14583 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.666875 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.666435 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.666893 14580 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.667922 13537 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.668875 13537 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.669801 13537 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.670962 15920 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.671228 15940 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.671480 15868 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.671900 15981 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.672374 15918 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.672477 15981 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.670841 15854 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.672887 15918 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.673211 13537 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.673761 15918 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.670897 15977 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.646588 15984 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.680725 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.681047 13657 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.681821 15973 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.668792 13769 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:07.674180 15909 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.682346 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:07.682436 13705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39134
I20260807 08:01:07.682642 15909 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.683051 15909 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.674712 15975 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.677726 15976 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.671543 15920 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.690639 15920 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.677847 14580 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.677879 13661 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.692651 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.692914 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.693177 14580 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.675037 14583 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.678759 15931 consensus_queue.cc:1046] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:07.677000 15854 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.679538 15926 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.698752 15926 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.698942 13661 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.699150 15926 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.699262 15879 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.674656 13537 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.683079 13705 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:07.686693 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.693825 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.700340 13661 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.694027 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.700515 13705 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.685860 13657 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.695248 14580 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.696041 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.702482 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.701256 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.696159 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:07.707705 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.703533 13769 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:07.708485 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.710170 14626 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42236
I20260807 08:01:07.710758 14626 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:07.696539 15902 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.696888 15921 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.697714 15976 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.694630 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.701862 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.702246 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.699718 15879 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:07.704770 15973 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.705224 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.705382 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.706354 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.710870 14583 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.711131 14626 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.710026 15922 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.706921 15855 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.712927 15854 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.681885 15986 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:07.714428 15931 raft_consensus.cc:3032] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:07.715415 13657 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:07.725932 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 51346.8us
I20260807 08:01:07.714704 15911 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.704113 14580 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.702764 15927 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.721077 13661 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.729363 13769 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.724862 14583 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.729142 13537 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.720750 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:07.729363 13657 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:07.717474 15958 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.748173 13706 raft_consensus.cc:1226] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:07.722498 15879 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.756268 13769 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.756067 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.732134 15927 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:07.757112 15921 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.757136 15893 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.757730 15893 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.758118 15893 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.759124 14583 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.726840 15976 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.723737 15868 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.761802 13537 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.761672 13769 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.766007 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.767540 13705 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:07.771687 14629 raft_consensus.cc:1226] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:07.729840 15931 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:07.729887 15975 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.733651 13661 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.770318 13657 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.733738 15854 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.773661 14629 raft_consensus.cc:1226] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:07.731675 15973 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.769022 13537 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.774959 13587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56594
I20260807 08:01:07.779834 14630 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:07.782431 13587 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:07.783130 13587 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:01:07.779170 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 45648.4us
I20260807 08:01:07.785010 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:07.774909 15992 log.cc:924] Time spent T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.052s user 0.000s sys 0.006s
I20260807 08:01:07.766434 15922 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.766804 15895 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.769572 15947 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.755574 15855 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.732745 15877 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.773738 15911 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.790890 14626 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.791356 13537 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.786496 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.757261 15927 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.789445 15904 consensus_queue.cc:1046] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260807 08:01:07.788167 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:07.788884 13705 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.770084 15981 raft_consensus.cc:2726] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:07.780840 13580 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 08:01:07.796108 15904 raft_consensus.cc:3032] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:07.791356 15940 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.796499 15904 raft_consensus.cc:729] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:07.781551 13700 raft_consensus.cc:1226] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:07.786798 15952 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.781431 15909 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.793778 15889 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.790895 15973 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.785808 15868 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.793875 13537 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.794457 13808 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:07.780143 14627 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:07.789781 15947 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.814181 15947 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.780407 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:07.816816 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:07.810913 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.811667 15909 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.825002 13706 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:07.778831 15976 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.814528 15911 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.796893 15940 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.835703 15940 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.836072 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:07.826169 13537 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.790010 15922 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.842953 13537 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.828783 14626 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:07.794668 14620 raft_consensus.cc:1226] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:07.825807 15931 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.835691 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.825088 13661 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.849135 13537 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.850244 13537 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.790284 15984 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.851210 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.854041 13537 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.855024 13537 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:07.850247 13661 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.828732 13579 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:07.856379 13809 raft_consensus.cc:2379] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:07.856632 13537 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.856866 13661 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.864599 15984 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.864840 15911 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.870853 15911 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.871218 15911 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.860693 13587 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.839639 15978 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:07.873037 13661 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.847535 15922 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.876109 13587 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.877199 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:07.884120 14626 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.847677 15909 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.885901 15870 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.862965 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.881736 15984 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.886523 13808 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.886778 15986 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:07.880414 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 103820us
I20260807 08:01:07.887778 13661 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:07.855625 15969 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:07.888098 14583 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.853204 15973 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.889024 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:07.855803 15977 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.855813 15897 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.861649 15901 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:07.897749 15976 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.898020 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.905821 13817 raft_consensus.cc:3032] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:07.906127 13817 raft_consensus.cc:729] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:07.823302 15904 consensus_queue.cc:260] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:07.848588 15952 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260807 08:01:07.901087 15978 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:07.879238 15975 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.896520 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.906853 13817 consensus_queue.cc:260] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.887378 15986 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.911175 13700 raft_consensus.cc:1226] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:07.865743 15926 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.933553 15926 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:07.911835 15899 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:07.890341 15931 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.917984 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:07.911872 15981 consensus_queue.cc:1046] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260807 08:01:07.921582 14626 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:07.892436 15969 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:07.895838 15907 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:07.925447 13817 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.936304 13769 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.936744 13540 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:01:07.938180 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 49239us
I20260807 08:01:07.907024 15901 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:07.915055 15973 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:07.942561 13661 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.925801 15978 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:07.945194 13661 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.946534 13661 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.947621 13661 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:07.948612 13661 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:07.949847 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.907672 15909 consensus_queue.cc:1046] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260807 08:01:07.956447 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:07.957305 13579 raft_consensus.cc:1226] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:07.954934 15877 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.951653 14583 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.958513 15976 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.952517 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:07.940810 15969 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:07.954180 15921 raft_consensus.cc:2726] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.954397 14620 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 LEARNER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:07.953698 15899 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:07.962337 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:07.962517 14626 raft_consensus.cc:1226] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:07.944742 15901 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:07.963711 13769 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:07.945130 15926 raft_consensus.cc:2726] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:07.954953 15868 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.951853 15984 raft_consensus.cc:2726] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.964243 13592 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260807 08:01:07.972267 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:07.956486 15889 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.939069 15999 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260807 08:01:08.010052 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.014063 13817 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:07.962742 16009 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:07.994419 15896 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.001322 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.011149 15909 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
W20260807 08:01:08.036759 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 100114us
I20260807 08:01:07.966427 15976 consensus_queue.cc:1046] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260807 08:01:08.024353 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:08.060902 13587 raft_consensus.cc:3032] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:08.061457 13587 raft_consensus.cc:729] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:08.040733 15909 raft_consensus.cc:686] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:08.074172 13587 consensus_queue.cc:260] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.071514 15947 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.078900 15920 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.079895 15893 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.080518 15947 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:08.081187 15920 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:08.081755 13812 raft_consensus.cc:1226] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.082573 13805 raft_consensus.cc:1226] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.075510 16008 consensus_queue.cc:1046] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260807 08:01:08.083027 13814 raft_consensus.cc:1226] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.092610 15904 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.114182 15909 consensus_queue.cc:237] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.120323 15969 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.121461 13589 raft_consensus.cc:3032] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.121867 13589 raft_consensus.cc:729] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:08.121897 13580 raft_consensus.cc:1226] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.124099 13589 consensus_queue.cc:260] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.125355 13589 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.082114 15893 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:08.134581 15940 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.135900 13579 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.136494 15879 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.116961 15978 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.139962 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:08.140340 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:08.127559 15918 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.142984 15918 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:08.143893 15975 raft_consensus.cc:3032] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:08.144258 15975 raft_consensus.cc:729] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:08.144663 14583 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.144884 15871 raft_consensus.cc:3032] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.145277 15871 raft_consensus.cc:729] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:08.145069 15975 consensus_queue.cc:260] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.146495 15975 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.143059 13710 raft_consensus.cc:1226] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.147389 13661 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.148818 14583 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.151504 13703 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.153092 16023 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.154093 16023 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:08.155275 15994 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.162055 15921 consensus_queue.cc:1046] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:08.162472 13812 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:08.165156 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 195259us
I20260807 08:01:08.165876 13708 raft_consensus.cc:1226] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:08.166443 13769 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.137382 15895 consensus_queue.cc:1046] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260807 08:01:08.167464 13702 raft_consensus.cc:2379] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:08.162367 15994 consensus_queue.cc:1046] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:08.156962 15976 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.168479 15921 raft_consensus.cc:3032] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.168849 15921 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:08.169093 15969 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:08.169711 15921 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.170800 15893 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.170974 15921 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.171937 13769 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.109740 15899 consensus_queue.cc:1046] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260807 08:01:08.176286 15899 raft_consensus.cc:3032] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.176411 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:08.173219 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.174224 15947 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.175082 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:08.178079 15947 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.173202 13808 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:08.176649 15899 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:08.179437 13769 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.181268 13769 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:08.184298 15920 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:08.180038 15899 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.184695 15920 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:08.185051 15920 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.186117 13657 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:08.186312 15940 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:08.188622 15952 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:08.189103 15952 raft_consensus.cc:686] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:08.190022 15952 consensus_queue.cc:237] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.191938 15895 consensus_queue.cc:1046] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:08.191956 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:08.193035 15895 consensus_queue.cc:1046] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:08.183974 13769 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.196221 13769 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.185305 15899 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.199064 13589 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:08.199301 13769 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.138801 15891 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.198405 15893 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.202589 15896 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.203477 15920 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.209746 13587 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.213347 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.214418 16015 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.219738 13769 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.217357 15897 consensus_queue.cc:1046] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:08.221000 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:08.221433 15920 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:08.222981 13769 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.182960 15947 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 08:01:08.217900 15954 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260807 08:01:08.260468 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:08.211184 15871 consensus_queue.cc:260] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.263036 15871 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.224768 15896 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.247808 13701 raft_consensus.cc:1226] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.243232 15909 raft_consensus.cc:3032] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:08.247182 15891 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:08.244642 15922 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.247897 15911 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.258967 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.274688 13769 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:08.274950 13587 raft_consensus.cc:2454] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:08.275992 14583 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.275983 13769 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.277119 16023 consensus_queue.cc:1046] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:08.277541 16015 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.277643 13769 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:08.278187 15885 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260807 08:01:08.278472 15933 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.259579 15920 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.279125 13591 raft_consensus.cc:1226] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.279786 15895 raft_consensus.cc:2781] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:08.279860 13769 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.271167 13814 raft_consensus.cc:1226] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.280647 15920 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.281641 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.282907 13769 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.283953 13769 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.284873 13540 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.222157 15978 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:08.273018 13661 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.277843 15911 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:08.284996 13769 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:08.267015 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 176874us
I20260807 08:01:08.259866 15893 raft_consensus.cc:2726] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.282168 15895 raft_consensus.cc:686] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:08.201480 15927 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.278192 16023 raft_consensus.cc:3032] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:08.288074 13537 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.299520 13769 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.268401 15909 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:08.285678 15922 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:08.315500 15909 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.316723 15909 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:08.317716 15954 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.285797 16024 raft_consensus.cc:2726] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.288331 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:08.287202 16008 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.285707 15902 consensus_queue.cc:1046] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260807 08:01:08.287362 15933 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.301270 15997 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260807 08:01:08.333618 15976 raft_consensus.cc:3032] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.333784 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:08.332093 15927 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:08.334785 13804 raft_consensus.cc:1226] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.301359 16015 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.335083 13816 raft_consensus.cc:1226] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:08.330550 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:08.329710 16010 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.336745 13805 raft_consensus.cc:1226] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:08.337380 15854 raft_consensus.cc:3032] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.337744 15854 raft_consensus.cc:729] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:08.334019 15976 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:08.283046 15947 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.325369 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:08.285966 15896 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:08.346596 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:08.348578 13537 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.350342 16010 consensus_queue.cc:1046] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:08.351042 15984 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:08.351922 15954 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:08.363059 15954 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: NON_VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: true } }
I20260807 08:01:08.364115 14620 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:08.346575 16001 consensus_queue.cc:1046] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260807 08:01:08.348299 15893 consensus_queue.cc:1046] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260807 08:01:08.366968 16001 raft_consensus.cc:3032] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:08.367228 16010 raft_consensus.cc:3032] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:08.367336 16001 raft_consensus.cc:729] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:08.367626 16010 raft_consensus.cc:729] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:08.367810 15879 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:08.368278 16001 consensus_queue.cc:260] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.301623 16003 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.348075 15972 consensus_queue.cc:1046] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260807 08:01:08.360108 15896 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:08.312774 16023 raft_consensus.cc:729] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:08.382372 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:08.383165 16023 consensus_queue.cc:260] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.384037 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:08.383929 15933 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:08.384356 13701 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 08:01:08.384747 16023 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.360574 16024 raft_consensus.cc:3032] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:08.361116 15986 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.360528 15895 consensus_queue.cc:237] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.368399 16010 consensus_queue.cc:260] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.395229 13588 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:08.356118 15947 consensus_queue.cc:1046] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:08.401078 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.401687 16001 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.371619 13540 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:08.377490 13431 catalog_manager.cc:5495] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:08.391193 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:08.391377 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:08.365305 15954 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:08.391942 14622 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.392379 14623 raft_consensus.cc:1226] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:08.392153 15854 consensus_queue.cc:260] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.392681 15920 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.388793 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:08.389272 16024 raft_consensus.cc:729] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:08.380177 16008 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.411446 13537 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.386143 15896 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:08.405421 16010 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.416733 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:08.417150 14627 raft_consensus.cc:2379] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:08.417042 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.403954 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.429276 13590 raft_consensus.cc:2379] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:08.429726 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.430599 13769 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:08.431703 13661 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.415580 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.432818 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:08.433241 13661 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.434068 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:08.434079 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.437168 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:08.437361 15947 raft_consensus.cc:2726] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.438405 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.439311 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.439474 16024 consensus_queue.cc:260] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.413573 15921 consensus_queue.cc:1046] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260807 08:01:08.440925 16024 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.440961 15921 raft_consensus.cc:3032] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.441438 15921 raft_consensus.cc:729] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:08.441635 15958 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.402500 15976 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.426262 15904 raft_consensus.cc:2726] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.417814 15972 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:08.444957 13537 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.458597 14630 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:08.401423 15901 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.461921 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.460738 13661 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.413328 15899 consensus_queue.cc:1046] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260807 08:01:08.464707 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.478318 13430 catalog_manager.cc:5495] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 2 to 3, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 3 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:08.481842 13585 raft_consensus.cc:1226] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:08.458240 15927 consensus_queue.cc:1046] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260807 08:01:08.458494 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:08.468905 15891 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:08.474397 15977 raft_consensus.cc:3032] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.468698 15854 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.471395 15972 raft_consensus.cc:686] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:08.460150 15952 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.468520 15918 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260807 08:01:08.516307 14620 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:08.518282 15972 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.499475 15977 raft_consensus.cc:729] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:08.494601 15921 consensus_queue.cc:260] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.505467 15891 raft_consensus.cc:686] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:08.525141 16008 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260807 08:01:08.526079 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:08.546226 15904 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.557891 15977 consensus_queue.cc:260] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.563443 13590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:08.564039 13590 raft_consensus.cc:595] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:08.565804 15893 raft_consensus.cc:3032] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:08.566179 15893 raft_consensus.cc:729] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:08.564383 13590 raft_consensus.cc:603] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:08.571837 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:08.571805 15893 consensus_queue.cc:260] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.560029 15921 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.562155 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:08.582234 14626 raft_consensus.cc:2932] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:08.589883 15997 raft_consensus.cc:3032] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.590301 15997 raft_consensus.cc:729] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:08.591060 15997 consensus_queue.cc:260] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.592253 15997 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.597720 15973 raft_consensus.cc:2932] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:08.641659 15994 raft_consensus.cc:3032] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.642153 15994 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:08.667292 15994 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.577862 15893 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.567956 15901 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:08.679220 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 67869.8us
I20260807 08:01:08.588932 14618 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:08.688297 13817 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:08.689399 13817 raft_consensus.cc:595] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:08.691293 13817 raft_consensus.cc:603] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260807 08:01:08.695434 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 46280.4us
I20260807 08:01:08.717164 13537 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.577759 15891 consensus_queue.cc:237] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.692394 15994 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.688292 13579 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:08.729197 13432 catalog_manager.cc:5495] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:08.734537 15931 raft_consensus.cc:2726] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:08.741124 13585 raft_consensus.cc:1261] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:08.745029 15976 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.749765 13585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:08.750495 13585 raft_consensus.cc:595] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.754207 13708 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:08.752652 15879 consensus_queue.cc:1046] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260807 08:01:08.758644 15889 raft_consensus.cc:3032] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:08.759016 15889 raft_consensus.cc:729] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:08.762497 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 266640us
I20260807 08:01:08.762795 13817 raft_consensus.cc:2932] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:08.780588 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 08:01:08.783298 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:08.759850 15889 consensus_queue.cc:260] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.789656 16010 consensus_queue.cc:1046] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:08.790335 15889 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.784929 13769 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:08.800149 16038 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:08.803566 16038 raft_consensus.cc:686] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:08.807796 13805 raft_consensus.cc:1226] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:08.807642 13769 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:08.804540 16038 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.809948 15897 raft_consensus.cc:979] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:08.810494 15897 raft_consensus.cc:1067] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Signalling peer 9210dbc4cc9c45e5b26a463b705d362c to start an election
W20260807 08:01:08.802942 16035 log.cc:924] Time spent T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260807 08:01:08.821285 13705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39134
I20260807 08:01:08.828840 13705 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:08.829173 13705 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.830298 14629 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:08.816991 13769 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:08.816511 16043 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.821478 13808 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:08.800740 16046 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:08.817281 15977 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.863205 13809 raft_consensus.cc:1261] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:08.859164 13769 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:08.853400 16043 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:08.879292 13705 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:08.902745 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 70982.1us
I20260807 08:01:08.816613 16016 consensus_queue.cc:1046] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20260807 08:01:08.867867 16024 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:08.858595 16046 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:08.921927 13705 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:08.841687 15897 consensus_queue.cc:1046] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
W20260807 08:01:08.926561 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 61854.4us
I20260807 08:01:08.862507 15981 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260807 08:01:08.935117 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:08.935983 13817 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.940059 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:08.940893 13590 raft_consensus.cc:3032] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:08.914582 16043 raft_consensus.cc:686] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:08.941265 13590 raft_consensus.cc:729] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:08.947182 14626 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:08.952890 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:08.971360 16046 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:08.922384 15986 consensus_queue.cc:1035] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260807 08:01:08.957969 13590 consensus_queue.cc:260] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:08.975003 13590 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:08.977954 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:08.978977 14627 raft_consensus.cc:2379] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 2: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:08.980665 13661 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:08.983757 16063 consensus_queue.cc:1035] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260807 08:01:09.010602 15896 consensus_meta.cc:287] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.575s user 0.005s sys 0.006s
I20260807 08:01:09.011099 15896 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:09.018572 13808 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:09.009341 16053 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:08.977535 14619 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:09.013357 15896 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:09.027935 13817 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:09.029410 13657 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:09.031158 13590 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:09.042989 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:08.994253 15885 raft_consensus.cc:3032] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:09.051008 15999 raft_consensus.cc:3032] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:09.048084 15921 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260807 08:01:09.050491 16001 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:09.065943 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:09.056329 15885 raft_consensus.cc:729] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:09.073232 13769 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:09.074370 16016 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:09.074885 16016 raft_consensus.cc:686] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:09.075536 13769 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:09.060489 15999 raft_consensus.cc:729] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:09.068914 16001 raft_consensus.cc:686] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:09.080802 16015 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:09.098353 15999 consensus_queue.cc:260] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.099829 15999 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:09.056253 16043 consensus_queue.cc:237] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.075775 16016 consensus_queue.cc:237] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:09.144686 13814 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:09.148315 16026 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:09.147058 13590 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:09.157752 16001 consensus_queue.cc:237] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:09.173669 16050 log.cc:924] Time spent T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.329s user 0.004s sys 0.001s
I20260807 08:01:09.184118 15918 raft_consensus.cc:3032] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:09.187955 14627 raft_consensus.cc:1226] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:09.202378 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:09.191113 15918 raft_consensus.cc:729] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:09.210216 16010 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:09.218118 16010 raft_consensus.cc:686] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:09.204236 15918 consensus_queue.cc:260] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.237742 13580 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:09.219175 16010 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.358492 14620 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:09.374495 13808 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:09.374872 16053 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:09.347146 15918 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:01:09.422900 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 44092.1us
I20260807 08:01:09.429113 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 550) took 1021 ms. Trace:
W20260807 08:01:09.429032 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 49427.2us
W20260807 08:01:09.514065 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 82479.4us
I20260807 08:01:09.429571 14629 rpcz_store.cc:276] 0807 08:01:08.407491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:08.407611 (+ 120us) service_pool.cc:227] Handling call
0807 08:01:08.432694 (+ 25083us) spinlock_profiling.cc:243] Waited 2.41 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:09.428904 (+996210us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b5400680454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:09.429055 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":729087744}
W20260807 08:01:09.563802 16084 log.cc:924] Time spent T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260807 08:01:09.526114 15994 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.574036 15994 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.576658 15994 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:09.555895 16063 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:09.578485 16063 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.580447 16063 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:09.432194 15909 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.123s
W20260807 08:01:09.662456 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 51666.4us
I20260807 08:01:09.416568 15885 consensus_queue.cc:260] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:09.557441 13579 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:09.671869 16026 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.689841 13661 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:09.686798 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:09.704067 13702 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:09.712922 15885 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:09.713629 15914 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.714124 15914 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.705734 16053 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:09.720207 16053 raft_consensus.cc:686] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:09.720805 15914 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:09.721334 15914 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.721086 16053 consensus_queue.cc:237] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:09.721772 15914 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.725458 15914 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:09.682958 16026 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:09.755431 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 174118us
I20260807 08:01:09.734256 15914 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.751318 15975 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:09.234323 15997 consensus_queue.cc:1159] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "22ff0a1f4a4241d794ab182432c85893" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 08:01:09.785898 13808 raft_consensus.cc:1261] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:09.766268 15914 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.774809 15975 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:09.838027 16026 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:09.838800 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:09.839349 15975 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:09.839555 13702 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
W20260807 08:01:09.863983 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 281845us
I20260807 08:01:09.863276 16010 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.896387 16010 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.895042 15975 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:09.895277 16026 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.895464 16023 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.897426 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:09.890462 16003 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:09.897255 16101 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.903668 15918 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:09.973764 16010 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:09.911032 15958 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:09.901315 15975 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.924134 16003 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:09.986598 16083 log.cc:924] Time spent T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260807 08:01:09.995949 13579 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:09.996737 16003 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:09.998236 16086 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.999091 16086 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.969129 16101 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.006937 15914 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.006296 16101 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.008029 16101 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:10.008888 16101 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.961340 15891 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:10.011477 15994 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.012014 15891 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.012344 15994 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.012720 16101 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.016755 15994 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:10.017804 15994 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.902480 16023 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.024102 13814 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:09.901928 16026 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:09.958037 16070 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.977857 15893 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:09.954368 15978 consensus_queue.cc:1035] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260807 08:01:10.035701 15914 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.041302 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:10.042287 14626 raft_consensus.cc:2379] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 2: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:10.040506 15914 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.033711 16070 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.035740 15893 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.048967 15893 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:10.049122 16070 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.051527 15914 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:10.055666 15902 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:10.033931 15885 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:10.059125 15885 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.056427 15902 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.026965 15891 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:09.970788 15918 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.018497 15994 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.075146 15909 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.076112 15909 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.077666 15975 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.078085 15918 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:10.079761 15909 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.080597 16023 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.080884 15885 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.084218 16026 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.019156 15958 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.229969 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.230829 13587 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:10.224346 13589 raft_consensus.cc:2932] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:10.241722 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.167253 15879 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:10.247435 16068 log.cc:924] Time spent T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260807 08:01:10.219722 16108 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.283537 15902 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.283836 15909 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.251621 15879 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.289306 15975 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:10.247300 16098 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.286430 15958 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.281970 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:10.246193 16021 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:10.286052 15994 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.270864 16026 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.297051 15904 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:10.305545 15975 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.314468 16120 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:10.317013 14583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.021s sys 0.025s
W20260807 08:01:10.331141 14583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.022s sys 0.026s
I20260807 08:01:10.310010 16098 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.315507 16003 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.316716 16026 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.335505 16098 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.330842 15904 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.336519 16026 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.338020 15904 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.294978 16108 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.334215 16003 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.341475 14583 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:10.349483 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1544) took 1210 ms. Trace:
I20260807 08:01:10.354326 13591 rpcz_store.cc:276] 0807 08:01:09.138683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:09.140879 (+ 2196us) service_pool.cc:227] Handling call
0807 08:01:09.147222 (+ 6343us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:09.147526 (+ 304us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:09.147553 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:09.148080 (+ 527us) log.cc:842] Serialized 26 byte log entry
0807 08:01:09.726073 (+577993us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b18002f50c0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a025345 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a042e22 00007f6b1cbec5bb 00007f6b1cbf2063 00007f6b1cbf2411 00007f6b1cf3f5cf
0807 08:01:10.019228 (+293155us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:10.019279 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:10.019372 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:10.341927 (+322555us) raft_consensus.cc:1671] finished
0807 08:01:10.341957 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:10.349393 (+ 7436us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":103,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":7038,"spinlock_wait_cycles":8132480,"thread_start_us":292952,"threads_started":1,"wal-append.queue_time_us":1149255}
I20260807 08:01:10.349261 16009 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:10.360806 14627 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:10.365451 13431 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56582 (request call id 303) took 1993 ms. Trace:
I20260807 08:01:10.357365 16003 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.302366 15909 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.384706 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.390079 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.390431 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.391108 13591 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:10.391804 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.392513 13585 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:10.386198 15921 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:10.383874 15879 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.349376 16108 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:10.396167 13769 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:10.401283 13540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.000s sys 0.076s
W20260807 08:01:10.411914 13540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.594s user 0.000s sys 0.076s
I20260807 08:01:10.401508 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.409469 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:10.413264 14626 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
W20260807 08:01:10.412258 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 594856us
I20260807 08:01:10.317670 16021 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.420746 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:10.421530 13709 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:10.426561 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:10.427273 13709 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:01:10.429062 16123 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.400976 15921 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:10.443745 13540 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:10.398839 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.435770 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:10.449180 13708 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:01:10.435067 14629 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:10.440979 16121 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:10.365063 16120 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.398222 16070 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:10.439200 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:10.408608 15902 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.446290 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:10.455608 13585 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:10.458334 13705 raft_consensus.cc:2379] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 2: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:10.464028 13703 raft_consensus.cc:2379] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:10.465274 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:10.466106 14630 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:10.432044 15893 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:10.429765 15904 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.432986 15918 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:10.476380 13592 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:10.448413 15986 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.448700 15921 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:10.441941 16098 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:10.365913 13431 rpcz_store.cc:276] 0807 08:01:08.350056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:08.350220 (+ 164us) service_pool.cc:227] Handling call
0807 08:01:10.343070 (+1992850us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:08.642629 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:08.658050 (+ 15421us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:08.658084 (+ 34us) write_op.cc:435] Acquired schema lock
0807 08:01:08.658102 (+ 18us) tablet.cc:660] Decoding operations
0807 08:01:08.682635 (+ 24533us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:08.688330 (+ 5695us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:08.690440 (+ 2110us) tablet.cc:683] Acquiring locks for 1 operations
0807 08:01:08.718448 (+ 28008us) tablet.cc:699] Row locks acquired
0807 08:01:08.720731 (+ 2283us) write_op.cc:260] PREPARE: finished
0807 08:01:08.733954 (+ 13223us) write_op.cc:270] Start()
0807 08:01:08.738572 (+ 4618us) write_op.cc:276] Timestamp: P: 1786089668732423 usec, L: 0
0807 08:01:08.751182 (+ 12610us) op_driver.cc:331] REPLICATION: starting
0807 08:01:08.815228 (+ 64046us) log.cc:842] Serialized 536 byte log entry
0807 08:01:09.368284 (+553056us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b1800329f20. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1cbfa82c 00007f6b1a0259dc 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a042e22 00007f6b1cbec5bb 00007f6b1cbf2063 00007f6b1cbf2411
0807 08:01:09.381228 (+ 12944us) op_driver.cc:444] REPLICATION: finished
0807 08:01:09.878263 (+497035us) write_op.cc:301] APPLY: starting
0807 08:01:09.880530 (+ 2267us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:09.893467 (+ 12937us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:09.895665 (+ 2198us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:09.909103 (+ 13438us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:09.913788 (+ 4685us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 08:01:09.923985 (+ 10197us) write_op.cc:312] APPLY: finished
0807 08:01:10.008471 (+ 84486us) log.cc:842] Serialized 23 byte log entry
0807 08:01:10.008648 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:10.008758 (+ 110us) write_op.cc:454] Released schema lock
0807 08:01:10.008851 (+ 93us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:10.009331 (+ 480us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:10.009372 (+ 41us) write_op.cc:454] Released schema lock
0807 08:01:10.051805 (+ 42433us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54000b0c54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf68689 00007f6b1cf06804 00007f6b1cf067b3 00007f6b1cf06728 00007f6b1cf066e3 00007f6b1cf059cc 00007f6b1cf2fa16 00007f6b1cf175cb 00007f6b1cf01c8d 00007f6b1cf05665
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":488457,"apply.run_cpu_time_us":3174,"apply.run_wall_time_us":136202,"num_ops":1,"prepare.queue_time_us":233388,"prepare.run_cpu_time_us":15817,"prepare.run_wall_time_us":726121,"raft.queue_time_us":550204,"raft.run_cpu_time_us":105738,"raft.run_wall_time_us":895298,"replication_time_us":629942,"spinlock_wait_cycles":502217728,"thread_start_us":648646,"threads_started":4,"wal-append.queue_time_us":87949}]]}
I20260807 08:01:10.451081 16117 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:10.598937 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:10.481714 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 53815.3us
I20260807 08:01:10.605959 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:10.492172 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:10.608981 13814 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
W20260807 08:01:10.493966 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 50303.6us
W20260807 08:01:10.547556 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 50521.6us
I20260807 08:01:10.489296 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:10.606802 13807 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:10.630533 16102 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.631162 16102 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.635391 13590 raft_consensus.cc:1261] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260807 08:01:10.600219 16016 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.014s sys 0.062s
W20260807 08:01:10.659854 16016 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s user 0.014s sys 0.064s
W20260807 08:01:10.680744 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 78579.5us
I20260807 08:01:10.491885 16121 raft_consensus.cc:686] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:10.497164 14625 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:10.493496 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.684746 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:10.599479 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:10.600159 15914 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.604497 15922 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:10.501490 16023 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:10.470817 15902 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.460436 16070 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.493705 15896 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.194s
W20260807 08:01:10.735466 13768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.010s sys 0.026s
I20260807 08:01:10.656212 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:10.494699 15918 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.599886 16098 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.494030 15904 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:10.757969 13768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.901s user 0.011s sys 0.026s
I20260807 08:01:10.698935 16026 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:10.698158 15973 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.633750 16113 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:10.751771 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:10.497597 15986 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.686380 15975 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.698662 16108 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:10.492515 15893 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.713799 16016 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:10.788779 16016 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.732684 16102 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:10.764945 16034 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:10.746722 16023 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.737068 15976 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:10.739228 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 57660.1us
I20260807 08:01:10.750828 15911 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.758520 15909 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.463299 16123 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.740518 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:10.808236 13807 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:10.734853 15914 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.628734 16117 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.761166 16026 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:10.765810 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:10.794303 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:10.802513 13769 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:10.739764 15922 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.779923 15975 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.810949 16095 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.812062 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:10.770746 15973 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.814556 15904 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.788313 16108 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:10.769726 14623 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:10.823194 13811 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:01:10.786078 13580 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:10.806031 15976 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.739349 16122 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20260807 08:01:10.806890 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:10.806416 16021 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.790795 16113 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.839968 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:10.885082 13807 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:10.849852 16108 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:10.857440 16067 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:10.866304 16136 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:10.870137 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 67767.2us
I20260807 08:01:10.871599 16122 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:10.856691 16095 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.872443 16086 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.823374 16132 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.835024 14625 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:01:10.825696 16026 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:10.861080 15976 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:10.793725 16109 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.435s
I20260807 08:01:10.921003 16123 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.873512 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:10.887766 16137 log.cc:924] Time spent T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260807 08:01:10.882586 15907 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:10.889156 15914 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.893287 13769 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:10.944868 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 51676us
I20260807 08:01:10.945742 13769 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:10.820118 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:10.906987 16151 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:10.917440 15896 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.920367 15976 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.817972 15911 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.925711 13812 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:10.834391 15975 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.896047 16122 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:10.834810 16034 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.934625 16095 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.825371 16121 consensus_queue.cc:237] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.816469 16125 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:10.899474 16120 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.944901 16070 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.950484 16130 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:10.950726 15986 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.953347 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:10.986321 16095 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:10.912797 16136 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.946833 15918 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:10.956435 16098 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:10.997885 16129 log.cc:924] Time spent T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.360s user 0.000s sys 0.001s
I20260807 08:01:10.998605 16117 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.973071 15902 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:10.999437 15911 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.954478 16104 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260807 08:01:10.977203 16145 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:11.007902 13706 raft_consensus.cc:2379] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:10.912249 16067 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:10.979743 16122 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:10.957108 15907 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:10.963634 15976 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:11.001129 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:10.964447 15896 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:11.010305 16095 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:11.008675 16023 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:10.944666 16132 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:11.032542 15918 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:11.061481 13430 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56596 (request call id 305) took 2593 ms. Trace:
I20260807 08:01:11.019675 16130 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:11.040077 15973 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:11.073626 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:11.040447 15922 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:11.006049 16125 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:11.078080 14625 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:11.082187 13811 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:11.044775 16145 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:11.061328 16095 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:10.980778 16151 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:11.069052 15918 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:11.099852 15973 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:11.111591 15952 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:11.124929 15952 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:11.076224 16034 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:11.108987 15918 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:11.109965 16023 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:11.156910 13657 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:11.159108 13657 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:11.065752 13430 rpcz_store.cc:276] 0807 08:01:08.461996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:08.462206 (+ 210us) service_pool.cc:227] Handling call
0807 08:01:11.055140 (+2592934us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:09.368836 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:09.369358 (+ 522us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:09.369386 (+ 28us) write_op.cc:435] Acquired schema lock
0807 08:01:09.369403 (+ 17us) tablet.cc:660] Decoding operations
0807 08:01:09.369829 (+ 426us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:09.369885 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:09.369918 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0807 08:01:09.372360 (+ 2442us) tablet.cc:699] Row locks acquired
0807 08:01:09.372378 (+ 18us) write_op.cc:260] PREPARE: finished
0807 08:01:09.372515 (+ 137us) write_op.cc:270] Start()
0807 08:01:09.372614 (+ 99us) write_op.cc:276] Timestamp: P: 1786089669372476 usec, L: 0
0807 08:01:09.372628 (+ 14us) op_driver.cc:331] REPLICATION: starting
0807 08:01:09.693024 (+320396us) log.cc:842] Serialized 541 byte log entry
0807 08:01:10.052398 (+359374us) op_driver.cc:444] REPLICATION: finished
0807 08:01:10.102694 (+ 50296us) write_op.cc:301] APPLY: starting
0807 08:01:10.104193 (+ 1499us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:10.104357 (+ 164us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:10.104518 (+ 161us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:10.104802 (+ 284us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:10.105036 (+ 234us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 08:01:10.105590 (+ 554us) write_op.cc:312] APPLY: finished
0807 08:01:10.105976 (+ 386us) log.cc:842] Serialized 23 byte log entry
0807 08:01:10.106234 (+ 258us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:10.106383 (+ 149us) write_op.cc:454] Released schema lock
0807 08:01:10.106442 (+ 59us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:10.112563 (+ 6121us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:10.144807 (+ 32244us) write_op.cc:454] Released schema lock
0807 08:01:10.314195 (+169388us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54000b0c54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf67198 00007f6b2097aade 00007f6b2097a2b8 00007f6b2097eba1 00007f6b2097eb59 00007f6b2097eae9 00007f6b2097eab1 00007f6b2097ddad 00007f6b232a7204 00007f6b232a7039
Metrics: {"mutex_wait_us":235666,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":14368,"apply.run_cpu_time_us":3138,"apply.run_wall_time_us":54727,"num_ops":1,"prepare.queue_time_us":887715,"prepare.run_cpu_time_us":17922,"prepare.run_wall_time_us":945627,"replication_time_us":679631,"spinlock_wait_cycles":734835584}]]}
I20260807 08:01:11.144359 15933 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:11.147974 16113 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:11.167438 13657 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:11.168365 13540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.072s sys 0.125s
W20260807 08:01:11.168757 13540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.072s sys 0.125s
W20260807 08:01:11.169049 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 559177us
I20260807 08:01:11.170584 13657 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:11.163497 15986 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:11.119014 15973 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:11.175602 13540 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:11.156606 16009 consensus_meta.cc:287] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.798s user 0.009s sys 0.007s
I20260807 08:01:11.151746 16034 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:11.153280 16151 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:11.167210 16070 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:11.208405 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:11.209349 13705 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:11.214010 13540 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:11.215402 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:11.159543 16130 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:11.220352 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:11.171952 15975 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:11.173589 15896 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:11.177903 15986 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:11.152370 16023 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:11.201478 16098 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:11.247139 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:11.255626 13706 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
W20260807 08:01:11.256536 14583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.024s sys 0.043s
I20260807 08:01:11.257416 13540 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:11.199972 15952 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:11.265247 13661 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.000s sys 0.017s
I20260807 08:01:11.218709 16070 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:11.223605 15907 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:11.171763 15933 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:11.210835 16034 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:11.246480 15986 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:11.358503 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:11.592573 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:11.987617 13661 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.377s user 0.018s sys 0.060s
I20260807 08:01:11.233379 16125 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:12.017972 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1518) took 1503 ms. Trace:
W20260807 08:01:11.984673 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 1.03896e+06us
I20260807 08:01:11.242625 16130 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 08:01:11.402537 13538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.032s sys 0.110s
W20260807 08:01:11.269492 14583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.652s user 0.025s sys 0.043s
W20260807 08:01:12.071156 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 1.58185e+06us
I20260807 08:01:12.070847 13817 rpcz_store.cc:276] 0807 08:01:10.483753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.486631 (+ 2878us) service_pool.cc:227] Handling call
0807 08:01:10.607475 (+120844us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:11.096231 (+488756us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:11.096276 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:11.096341 (+ 65us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:11.096381 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:11.987310 (+890929us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54001fbdd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:11.987355 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:11.987475 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1123474688}
I20260807 08:01:11.598878 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:12.043226 13706 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
W20260807 08:01:11.401919 16134 log.cc:924] Time spent T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
W20260807 08:01:12.086494 13538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.424s user 0.038s sys 0.138s
I20260807 08:01:11.264533 16098 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:12.074011 14583 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:11.458992 15981 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
W20260807 08:01:11.476019 16096 log.cc:924] Time spent T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 1.489s user 0.053s sys 0.063s
W20260807 08:01:12.042503 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 1.43194e+06us
I20260807 08:01:12.024094 16010 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:11.250757 16132 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:11.318615 15911 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:12.109992 15897 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:12.119979 15911 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:12.120658 15911 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:12.121176 16142 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:12.120025 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:11.254539 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:11.263803 15976 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:12.043323 15921 consensus_meta.cc:287] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 1.416s user 0.014s sys 0.002s
I20260807 08:01:11.406136 16144 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:11.591997 13812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 220) took 1051 ms. Trace:
I20260807 08:01:11.268983 15922 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:12.082608 13809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1133) took 2159 ms (client timeout 2035 ms). Trace:
I20260807 08:01:11.319676 14580 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:11.408936 16136 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:12.130128 13708 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:11.981273 16150 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:11.939704 16135 log.cc:924] Time spent T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.399s user 0.000s sys 0.001s
I20260807 08:01:12.041416 14627 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:12.006542 16070 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:12.034433 16153 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:12.017776 16009 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:12.104010 14583 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:11.542475 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:12.045746 13808 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.014s sys 0.036s
I20260807 08:01:11.537770 14622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 563) took 1105 ms. Trace:
I20260807 08:01:12.032421 16152 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:11.410501 16081 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:11.270155 15902 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:12.042922 13540 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.561s (SENT) [suppressed 2 similar messages]
I20260807 08:01:12.110962 13661 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:11.586926 14623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 559) took 1156 ms. Trace:
I20260807 08:01:11.478819 15975 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:11.973655 15977 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:11.972697 15904 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:12.018927 15927 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:12.111778 13657 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.939s user 0.022s sys 0.027s
I20260807 08:01:11.356913 15914 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
W20260807 08:01:12.231237 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 127332us
W20260807 08:01:12.113509 16085 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s user 0.003s sys 0.001s
I20260807 08:01:12.032429 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:12.048087 16120 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:12.071204 16130 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:11.960661 16016 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:11.458763 16147 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:11.268728 16151 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:11.538219 15877 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:11.590878 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:12.348444 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 272287us
I20260807 08:01:12.112721 16154 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:12.112447 16149 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:12.117549 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:12.131700 15921 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.106776 16098 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:11.422938 16097 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:11.682996 16145 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:11.595618 14629 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:12.381206 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:12.033663 16159 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:12.086416 16117 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:11.596318 16158 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:12.093314 15973 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:12.114239 16023 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:12.134158 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 814541us
I20260807 08:01:12.121351 16102 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:12.119304 16125 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:11.963429 16139 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:12.120980 16010 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.114284 15981 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.120594 15897 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.122097 16142 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:12.126771 16053 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.355s user 0.057s sys 0.099s
W20260807 08:01:12.133448 13809 rpcz_store.cc:269] 0807 08:01:09.914950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:09.915101 (+ 151us) service_pool.cc:227] Handling call
0807 08:01:11.592364 (+1677263us) spinlock_profiling.cc:243] Waited 7.64 ms on lock 0x7b5800004298. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:12.073122 (+480758us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17564032}
I20260807 08:01:12.103657 16162 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:12.149137 16086 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:12.226491 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 115610us
W20260807 08:01:12.230650 13657 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.058s user 0.022s sys 0.028s
I20260807 08:01:12.231010 16124 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:12.129679 13702 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:12.150667 15997 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:12.138870 15976 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 08:01:12.225630 13808 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.029s user 0.014s sys 0.046s
I20260807 08:01:12.135191 15954 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:12.138113 16144 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:12.261925 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:12.220255 16015 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:12.233498 16026 consensus_meta.cc:287] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.302s user 0.016s sys 0.000s
I20260807 08:01:12.226389 15902 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:12.359935 13817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1139) took 1204 ms (client timeout 1561 ms). Trace:
I20260807 08:01:11.318641 16113 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:12.360531 13807 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.439s user 0.028s sys 0.065s
I20260807 08:01:12.138270 15922 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.372687 16172 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:12.264030 16130 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:12.233369 16148 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:12.262115 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 568) took 1657 ms. Trace:
I20260807 08:01:12.258967 15927 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.228820 16009 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.231694 15975 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.232036 16081 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.233901 15904 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.233008 15914 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.228309 16152 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.232852 15977 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.264974 16120 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.228018 16150 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.356132 16151 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.241592 16153 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.767417 13584 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:12.767148 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:12.514544 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:12.768642 13584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1541) took 2325 ms (client timeout 1869 ms). Trace:
W20260807 08:01:12.473464 13592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1562) took 2028 ms (client timeout 2422 ms). Trace:
I20260807 08:01:12.131378 13812 rpcz_store.cc:276] 0807 08:01:10.540193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.801421 (+261228us) service_pool.cc:227] Handling call
0807 08:01:11.591628 (+790207us) spinlock_profiling.cc:243] Waited 94.9 ms on lock 0x7b5400125a54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:11.591837 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":218187520}
W20260807 08:01:12.386030 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 154098us
I20260807 08:01:12.224546 14623 rpcz_store.cc:276] 0807 08:01:10.430441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.436103 (+ 5662us) service_pool.cc:227] Handling call
0807 08:01:11.586656 (+1150553us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b5400450454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:11.586818 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86676352}
I20260807 08:01:12.370185 15933 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:12.771868 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:12.704183 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:12.375386 16135 log.cc:924] Time spent T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20260807 08:01:12.764410 13540 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:12.769070 13584 rpcz_store.cc:269] 0807 08:01:10.443234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.261625 (+1818391us) service_pool.cc:227] Handling call
0807 08:01:12.768585 (+506960us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:13.332203 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 567775us
I20260807 08:01:13.332755 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:12.487413 13588 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:13.334370 14620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 380) took 3453 ms. Trace:
I20260807 08:01:13.241391 13585 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
W20260807 08:01:12.769814 13592 rpcz_store.cc:269] 0807 08:01:10.389697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.389817 (+ 120us) service_pool.cc:227] Handling call
0807 08:01:12.417932 (+2028115us) spinlock_profiling.cc:243] Waited 85.5 ms on lock 0x7b54002add54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:12.418058 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":196695424}
W20260807 08:01:13.241866 13591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33050 (request call id 230) took 2489 ms (client timeout 2152 ms). Trace:
W20260807 08:01:13.353818 13591 rpcz_store.cc:269] 0807 08:01:10.401896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.402073 (+ 177us) service_pool.cc:227] Handling call
0807 08:01:12.771727 (+2369654us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b5800001898. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:12.891286 (+119559us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4909952}
I20260807 08:01:13.354722 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.355383 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.355896 13592 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:13.356931 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.357121 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:13.243297 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 474399us
I20260807 08:01:12.480882 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1252) took 1212 ms. Trace:
I20260807 08:01:13.366007 13701 rpcz_store.cc:276] 0807 08:01:11.206882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:11.219602 (+ 12720us) service_pool.cc:227] Handling call
0807 08:01:11.590606 (+371004us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:12.419278 (+828672us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7168256}
W20260807 08:01:12.898312 13702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1251) took 1662 ms (client timeout 1624 ms). Trace:
I20260807 08:01:13.250690 13710 raft_consensus.cc:2379] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:13.251782 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1258) took 2009 ms. Trace:
W20260807 08:01:12.796712 13709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1559) took 2342 ms (client timeout 1869 ms). Trace:
W20260807 08:01:13.375174 13709 rpcz_store.cc:269] 0807 08:01:10.425645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.454704 (+ 29059us) service_pool.cc:227] Handling call
0807 08:01:12.766953 (+2312249us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:12.768274 (+ 1321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42985856}
I20260807 08:01:13.369680 13705 rpcz_store.cc:276] 0807 08:01:11.242369 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:11.242944 (+ 575us) service_pool.cc:227] Handling call
0807 08:01:13.011388 (+1768444us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:13.250024 (+238636us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:13.250747 (+ 723us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:13.250813 (+ 66us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.250907 (+ 94us) raft_consensus.cc:1631] Marking committed up to 1
0807 08:01:13.250997 (+ 90us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.251099 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.251400 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5644160}
W20260807 08:01:13.368394 13702 rpcz_store.cc:269] 0807 08:01:11.206522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:11.219655 (+ 13133us) service_pool.cc:227] Handling call
0807 08:01:12.868636 (+1648981us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:13.377841 14629 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.003s sys 0.022s
W20260807 08:01:13.378289 14629 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.838s user 0.003s sys 0.022s
W20260807 08:01:12.810653 13711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1248) took 2348 ms (client timeout 1561 ms). Trace:
I20260807 08:01:13.252964 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:13.414291 13711 rpcz_store.cc:269] 0807 08:01:10.419834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.420337 (+ 503us) service_pool.cc:227] Handling call
0807 08:01:12.514339 (+2094002us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:12.768304 (+253965us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3589376}
I20260807 08:01:13.414989 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1544) took 1057 ms. Trace:
I20260807 08:01:13.415498 13587 rpcz_store.cc:276] 0807 08:01:12.357791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.360832 (+ 3041us) service_pool.cc:227] Handling call
0807 08:01:12.361208 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:13.395782 (+1034574us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:13.413619 (+ 17837us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.413683 (+ 64us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:13.413774 (+ 91us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.414567 (+ 793us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54002ac954. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:13.414614 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.414804 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2247281920}
I20260807 08:01:13.416982 15977 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:12.797142 13807 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.852s user 0.031s sys 0.074s
W20260807 08:01:12.904363 13809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1142) took 1724 ms (client timeout 1624 ms). Trace:
I20260807 08:01:13.256055 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1522) took 2770 ms. Trace:
W20260807 08:01:13.444098 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 107963us
W20260807 08:01:12.767197 13817 rpcz_store.cc:269] 0807 08:01:11.155978 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.086882 (+930904us) service_pool.cc:227] Handling call
0807 08:01:12.359868 (+272986us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:13.256244 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 469945us
I20260807 08:01:12.757159 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:13.258596 13811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 217) took 2761 ms (client timeout 2692 ms). Trace:
W20260807 08:01:13.026909 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1147) took 1843 ms (client timeout 1665 ms). Trace:
W20260807 08:01:13.259155 14580 connection.cc:232] RPC call timeout handler was delayed by -1.1485662849154323s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
W20260807 08:01:13.451382 13812 rpcz_store.cc:269] 0807 08:01:11.183825 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.961105 (+1777280us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.022206 (+ 61101us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.452351 13769 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:13.452795 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1141) took 2272 ms (client timeout 1707 ms). Trace:
W20260807 08:01:13.452899 13817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1140) took 2296 ms (client timeout 1731 ms). Trace:
I20260807 08:01:12.718281 14619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 566) took 2277 ms. Trace:
I20260807 08:01:12.362774 15877 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.417243 15893 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:12.664440 15918 consensus_meta.cc:287] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 1.513s user 0.012s sys 0.008s
I20260807 08:01:13.460983 15918 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.462651 15877 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.463207 16151 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:13.463671 15918 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.463541 15920 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:12.767681 15976 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:13.464661 16126 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:13.464920 15976 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:12.705904 15986 consensus_meta.cc:287] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.619s user 0.010s sys 0.001s
I20260807 08:01:13.467593 16175 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:13.468212 15986 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:12.805401 15997 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:13.469686 16122 consensus_meta.cc:287] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 2.411s user 0.015s sys 0.018s
I20260807 08:01:12.796937 14627 rpcz_store.cc:276] 0807 08:01:10.602642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:11.477698 (+875056us) service_pool.cc:227] Handling call
0807 08:01:11.938859 (+461161us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:12.224834 (+285975us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:12.260299 (+ 35465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:12.798659 16015 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:12.796312 16053 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.004s user 0.064s sys 0.130s
W20260807 08:01:13.476908 16130 consensus_meta.cc:287] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.609s user 0.015s sys 0.007s
W20260807 08:01:12.415716 16108 consensus_meta.cc:287] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 1.511s user 0.011s sys 0.004s
I20260807 08:01:13.477639 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.477421 16130 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:13.477859 16108 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.478463 14627 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:13.479166 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 389) took 1222 ms. Trace:
I20260807 08:01:12.419256 16117 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.288426 15902 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:13.475291 16070 consensus_meta.cc:287] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.242s user 0.016s sys 0.033s
I20260807 08:01:12.780332 16026 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.480952 14626 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:13.481356 16193 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:13.480690 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 567) took 3039 ms. Trace:
I20260807 08:01:13.482077 15997 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:12.505050 16034 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:13.453297 13812 rpcz_store.cc:269] 0807 08:01:11.179982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.452532 (+2272550us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.452715 (+ 183us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.482452 16026 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:13.483047 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 382) took 3029 ms (client timeout 1903 ms). Trace:
W20260807 08:01:13.483443 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1135) took 2330 ms (client timeout 1801 ms). Trace:
W20260807 08:01:13.483982 13812 rpcz_store.cc:269] 0807 08:01:11.152922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.483225 (+2330303us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.483357 (+ 132us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.483794 16070 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:13.294751 16095 consensus_meta.cc:287] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 2.144s user 0.013s sys 0.009s
I20260807 08:01:13.294952 15897 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:12.384845 16097 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:13.295545 13432 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57868 (request call id 212) took 4611 ms. Trace:
I20260807 08:01:13.296572 15921 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:12.796607 16102 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:13.297375 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:10.355326 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:12.779052 16144 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:12.771492 16105 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:12.823241 16113 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.804975 16124 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.890357 16139 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:12.799063 15973 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:12.370472 16147 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.389143 16149 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.798435 16158 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:12.370008 16154 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:12.419713 16159 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:12.685320 16160 log.cc:924] Time spent T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.308s user 0.000s sys 0.001s
W20260807 08:01:12.505568 16166 log.cc:924] Time spent T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.349s user 0.001s sys 0.000s
I20260807 08:01:12.231556 16167 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:12.779639 16162 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:13.025321 16170 log.cc:924] Time spent T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.365s user 0.000s sys 0.001s
I20260807 08:01:13.310032 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.312915 16148 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:13.315302 14625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 386) took 2825 ms (client timeout 3050 ms). Trace:
I20260807 08:01:13.311841 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1560) took 2927 ms. Trace:
I20260807 08:01:12.222409 14622 rpcz_store.cc:276] 0807 08:01:10.432007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.447216 (+ 15209us) service_pool.cc:227] Handling call
0807 08:01:11.536608 (+1089392us) spinlock_profiling.cc:243] Waited 6.38 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00bbce 000000000034aea9 00007f6b21e37a30 00007f6b21efd7b4 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:11.536847 (+ 239us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:11.537109 (+ 262us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:11.537140 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:11.537185 (+ 45us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:11.537279 (+ 94us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:11.537403 (+ 124us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:11.537638 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14667136}
I20260807 08:01:13.299034 16120 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.319051 15904 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.312040 13428 catalog_manager.cc:5495] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 2 to 3, leader changed from 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:13.325450 15994 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:13.328361 16176 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:13.243484 13659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.000s sys 0.010s
I20260807 08:01:13.360214 13584 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:13.360507 16085 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:13.357818 13591 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:01:13.325397 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.334822 14620 rpcz_store.cc:276] 0807 08:01:09.880973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:09.890375 (+ 9402us) service_pool.cc:227] Handling call
0807 08:01:09.911517 (+ 21142us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:10.019870 (+108353us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:10.025418 (+ 5548us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:10.069908 (+ 44490us) log.cc:842] Serialized 26 byte log entry
0807 08:01:10.599032 (+529124us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:10.737991 (+138959us) log.cc:842] Serialized 14 byte log entry
0807 08:01:10.755610 (+ 17619us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:10.761710 (+ 6100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:11.540286 (+778576us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b18002e0b40. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:13.333712 (+1793426us) raft_consensus.cc:1671] finished
0807 08:01:13.333876 (+ 164us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.334134 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":77219,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":183624,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":17862,"spinlock_wait_cycles":8853888,"thread_start_us":418508,"threads_started":1,"wal-append.queue_time_us":680586}
I20260807 08:01:13.398819 15922 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:13.399555 13710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1246) took 2980 ms (client timeout 2167 ms). Trace:
I20260807 08:01:13.399698 15975 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.401387 16010 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:13.402117 13708 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.007s sys 0.016s
I20260807 08:01:13.401978 16086 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.403240 16152 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:13.403172 16125 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:13.405028 16150 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.406358 15927 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:13.407027 15911 consensus_meta.cc:287] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.285s user 0.020s sys 0.006s
W20260807 08:01:13.485062 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1152) took 2258 ms (client timeout 1753 ms). Trace:
I20260807 08:01:13.493867 13590 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:01:13.494392 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1549) took 1113 ms. Trace:
I20260807 08:01:13.494551 13590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1547) took 1114 ms. Trace:
I20260807 08:01:13.494778 13591 rpcz_store.cc:276] 0807 08:01:12.381115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.356608 (+975493us) service_pool.cc:227] Handling call
0807 08:01:13.494306 (+137698us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:13.495437 14625 rpcz_store.cc:269] 0807 08:01:10.489614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.945470 (+455856us) service_pool.cc:227] Handling call
0807 08:01:13.314716 (+2369246us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b54006ac2d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:13.315017 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28249472}
I20260807 08:01:13.495846 16162 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:13.495941 13710 rpcz_store.cc:269] 0807 08:01:10.418591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.419391 (+ 800us) service_pool.cc:227] Handling call
0807 08:01:12.703991 (+2284600us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:13.399369 (+695378us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12449536}
W20260807 08:01:13.497241 13708 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.963s user 0.008s sys 0.016s
I20260807 08:01:13.496786 16095 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.498466 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1561) took 3070 ms. Trace:
I20260807 08:01:13.499492 16095 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:13.492561 13809 rpcz_store.cc:269] 0807 08:01:11.180632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.904122 (+1723490us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:12.904279 (+ 157us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260807 08:01:13.501120 13809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1533) took 2322 ms (client timeout 1950 ms). Trace:
W20260807 08:01:13.501730 13809 rpcz_store.cc:269] 0807 08:01:11.178825 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.500933 (+2322108us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.501064 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.502306 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:13.502583 13809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1146) took 2319 ms (client timeout 1985 ms). Trace:
I20260807 08:01:13.503131 14625 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:13.503487 13584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1551) took 1121 ms. Trace:
I20260807 08:01:13.503557 16158 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:13.503710 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 391) took 1130 ms. Trace:
I20260807 08:01:13.503942 13584 rpcz_store.cc:276] 0807 08:01:12.382665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.332132 (+949467us) service_pool.cc:227] Handling call
0807 08:01:13.503321 (+171189us) spinlock_profiling.cc:243] Waited 9.7 ms on lock 0x7b540036a954. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:13.503433 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22306560}
I20260807 08:01:13.504511 16139 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:13.487993 16121 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:13.411665 16098 consensus_meta.cc:287] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.022s user 0.002s sys 0.024s
W20260807 08:01:13.414876 13700 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1247) took 2996 ms (client timeout 2216 ms). Trace:
I20260807 08:01:13.014531 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1530) took 2518 ms. Trace:
I20260807 08:01:13.412058 16172 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:13.399708 13657 connection.cc:232] RPC call timeout handler was delayed by -1.782784319948405s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1684s for a response
I20260807 08:01:13.402009 16142 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.433288 16067 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.417804 16195 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:13.434460 15981 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:13.434986 16168 log.cc:924] Time spent T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.113s user 0.003s sys 0.000s
W20260807 08:01:13.435014 13661 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.011s sys 0.012s
W20260807 08:01:13.439641 14582 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.005s sys 0.024s
I20260807 08:01:13.440804 16081 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.439101 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.445182 16198 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:13.441172 16009 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.450621 13804 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:13.447069 13816 rpcz_store.cc:276] 0807 08:01:10.485086 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.486997 (+ 1911us) service_pool.cc:227] Handling call
0807 08:01:10.487352 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:10.768974 (+281622us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:10.769012 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:10.769068 (+ 56us) raft_consensus.cc:1631] Marking committed up to 1
0807 08:01:10.769101 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.017831 (+2248730us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b54003c4bd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:13.255338 (+237507us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.255762 (+ 424us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":647172352}
W20260807 08:01:13.449040 13811 rpcz_store.cc:269] 0807 08:01:10.496922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.867314 (+370392us) service_pool.cc:227] Handling call
0807 08:01:13.007731 (+2140417us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540027b0d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:13.258291 (+250560us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":345050880}
I20260807 08:01:13.454025 14619 rpcz_store.cc:276] 0807 08:01:10.441080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.461629 (+ 20549us) service_pool.cc:227] Handling call
0807 08:01:12.704607 (+2242978us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:12.704962 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:12.705282 (+ 320us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:12.705344 (+ 62us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:12.705416 (+ 72us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:12.705474 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:12.706707 (+ 1233us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:12.718169 (+ 11462us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":62806912}
W20260807 08:01:13.453384 13817 rpcz_store.cc:269] 0807 08:01:11.156320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.452567 (+2296247us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.452727 (+ 160us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:12.805128 15954 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.554580 15954 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.465957 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1548) took 1078 ms. Trace:
I20260807 08:01:13.555226 16113 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.555297 13585 rpcz_store.cc:276] 0807 08:01:12.380828 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.380944 (+ 116us) service_pool.cc:227] Handling call
0807 08:01:13.447503 (+1066559us) spinlock_profiling.cc:243] Waited 859 ms on lock 0x7b54003d0ed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:13.458754 (+ 11251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1975399680}
I20260807 08:01:13.473817 16122 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.475215 15986 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.476207 13703 raft_consensus.cc:2379] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:13.478578 16015 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:13.479809 16130 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.480036 16108 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:13.287283 16003 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:12.536935 14618 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:13.291570 15914 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.501092 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.411109 16153 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:13.502962 13809 rpcz_store.cc:269] 0807 08:01:11.183279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.502410 (+2319131us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.502529 (+ 119us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.495884 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.505162 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:13.510012 13659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.780s user 0.000s sys 0.011s
W20260807 08:01:13.510210 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 45037us
I20260807 08:01:13.498848 13708 rpcz_store.cc:276] 0807 08:01:10.428190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.458198 (+ 30008us) service_pool.cc:227] Handling call
0807 08:01:12.129748 (+1671550us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:12.129917 (+ 169us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:12.130052 (+ 135us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 08:01:12.534114 (+404062us) log.cc:842] Serialized 14 byte log entry
0807 08:01:13.497848 (+963734us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.497979 (+ 131us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:13.498050 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.498188 (+ 138us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.498390 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5808512,"thread_start_us":963180,"threads_started":1,"wal-append.queue_time_us":900909}
W20260807 08:01:13.483517 14626 rpcz_store.cc:269] 0807 08:01:10.453905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.998505 (+544600us) service_pool.cc:227] Handling call
0807 08:01:11.598700 (+600195us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:13.482818 (+1884118us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54003354d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:13.482991 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3223325184}
I20260807 08:01:13.516778 15911 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:13.516885 16121 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.479575 14627 rpcz_store.cc:276] 0807 08:01:12.257138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.477045 (+1219907us) service_pool.cc:227] Handling call
0807 08:01:13.479089 (+ 2044us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.494917 13590 rpcz_store.cc:276] 0807 08:01:12.380509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.380659 (+ 150us) service_pool.cc:227] Handling call
0807 08:01:13.309485 (+928826us) spinlock_profiling.cc:243] Waited 4.02 ms on lock 0x7b5800001898. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:13.494497 (+185012us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9244032}
I20260807 08:01:13.501401 16086 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.504233 14625 rpcz_store.cc:276] 0807 08:01:12.373628 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.501941 (+1128313us) service_pool.cc:227] Handling call
0807 08:01:13.503642 (+ 1701us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.493790 16085 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:13.559614 14619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 388) took 1340 ms. Trace:
W20260807 08:01:13.521097 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 44647.3us
I20260807 08:01:13.507339 16193 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.533354 16154 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:13.534050 16124 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:13.505983 13812 rpcz_store.cc:269] 0807 08:01:11.226914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.484877 (+2257963us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.485006 (+ 129us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.537657 16167 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:13.538066 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1546) took 1175 ms. Trace:
I20260807 08:01:13.538335 16105 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.514276 16098 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:13.539521 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:13.545666 14582 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.779s user 0.005s sys 0.024s
I20260807 08:01:13.522841 15994 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:13.546118 13769 connection.cc:232] RPC call timeout handler was delayed by -1.3566105188801885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
W20260807 08:01:13.546005 13661 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.777s user 0.012s sys 0.012s
I20260807 08:01:13.527619 16102 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:13.482440 14629 rpcz_store.cc:276] 0807 08:01:10.441444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.999686 (+558242us) service_pool.cc:227] Handling call
0807 08:01:11.594982 (+595296us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:11.595210 (+ 228us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:11.595482 (+ 272us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 08:01:12.540366 (+944884us) log.cc:842] Serialized 14 byte log entry
0807 08:01:13.443650 (+903284us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.449147 (+ 5497us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:13.467223 (+ 18076us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.467286 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.480625 (+ 13339us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7849,"raft.queue_time_us":91350,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":299,"spinlock_wait_cycles":2976512,"thread_start_us":854244,"threads_started":2,"wal-append.queue_time_us":919209}
I20260807 08:01:13.544344 13805 rpcz_store.cc:276] 0807 08:01:10.496608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:11.012408 (+515800us) service_pool.cc:227] Handling call
0807 08:01:11.018444 (+ 6036us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:11.118035 (+ 99591us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:11.118083 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:11.118145 (+ 62us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:11.118189 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.014197 (+1896008us) spinlock_profiling.cc:243] Waited 99.3 ms on lock 0x7b54001b7454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:13.014258 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.014456 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":228369792}
I20260807 08:01:13.549019 16053 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:13.496196 13579 rpcz_store.cc:276] 0807 08:01:10.383943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.386276 (+ 2333us) service_pool.cc:227] Handling call
0807 08:01:12.358543 (+1972267us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b5800001898. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:12.384030 (+ 25487us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:13.309350 (+925320us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:13.309451 (+ 101us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.309572 (+ 121us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:13.309652 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.311061 (+ 1409us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b54001356d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:13.311172 (+ 111us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.311519 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3524,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":357,"spinlock_wait_cycles":2113714432}
W20260807 08:01:13.665695 13769 connection.cc:232] RPC call timeout handler was delayed by -1.3566105188801885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
I20260807 08:01:13.526551 16117 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:13.508253 13700 rpcz_store.cc:269] 0807 08:01:10.418895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.431159 (+ 12264us) service_pool.cc:227] Handling call
0807 08:01:13.011072 (+2579913us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:13.414778 (+403706us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27846912}
I20260807 08:01:13.556217 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1525) took 3067 ms. Trace:
I20260807 08:01:13.556985 13592 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:13.557569 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.536614 14623 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
W20260807 08:01:13.558801 13804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1523) took 3071 ms (client timeout 1684 ms). Trace:
W20260807 08:01:13.558969 13817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1531) took 2381 ms (client timeout 2079 ms). Trace:
I20260807 08:01:13.549201 16198 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:13.668537 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 225) took 2404 ms (client timeout 2093 ms). Trace:
W20260807 08:01:13.559474 13811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1151) took 2340 ms (client timeout 2126 ms). Trace:
I20260807 08:01:13.648100 16085 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:13.674660 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.506486 13432 rpcz_store.cc:276] 0807 08:01:08.684349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:08.684492 (+ 143us) service_pool.cc:227] Handling call
0807 08:01:13.295166 (+4610674us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:10.314876 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:10.333776 (+ 18900us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:10.333829 (+ 53us) write_op.cc:435] Acquired schema lock
0807 08:01:10.333873 (+ 44us) tablet.cc:660] Decoding operations
0807 08:01:10.334274 (+ 401us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:10.334323 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:10.334371 (+ 48us) tablet.cc:683] Acquiring locks for 1 operations
0807 08:01:10.334605 (+ 234us) tablet.cc:699] Row locks acquired
0807 08:01:10.334624 (+ 19us) write_op.cc:260] PREPARE: finished
0807 08:01:10.334784 (+ 160us) write_op.cc:270] Start()
0807 08:01:10.334923 (+ 139us) write_op.cc:276] Timestamp: P: 1786089670334733 usec, L: 0
0807 08:01:10.334954 (+ 31us) op_driver.cc:331] REPLICATION: starting
0807 08:01:10.335695 (+ 741us) log.cc:842] Serialized 536 byte log entry
0807 08:01:10.340918 (+ 5223us) op_driver.cc:444] REPLICATION: finished
0807 08:01:10.341544 (+ 626us) write_op.cc:301] APPLY: starting
0807 08:01:10.341635 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:10.341763 (+ 128us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:10.347490 (+ 5727us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:10.347800 (+ 310us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:10.347994 (+ 194us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 08:01:10.348048 (+ 54us) write_op.cc:312] APPLY: finished
0807 08:01:10.348262 (+ 214us) log.cc:842] Serialized 23 byte log entry
0807 08:01:10.348462 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:10.348573 (+ 111us) write_op.cc:454] Released schema lock
0807 08:01:10.348642 (+ 69us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:10.348936 (+ 294us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:10.348972 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":700317,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":305,"apply.run_cpu_time_us":2118,"apply.run_wall_time_us":7693,"num_ops":1,"prepare.queue_time_us":1581908,"prepare.run_cpu_time_us":3168,"prepare.run_wall_time_us":21157,"replication_time_us":5849}]]}
W20260807 08:01:13.560263 13816 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1534) took 2381 ms (client timeout 2066 ms). Trace:
W20260807 08:01:13.675835 13816 rpcz_store.cc:269] 0807 08:01:11.179530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.549840 (+2370310us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.560183 (+ 10343us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.657744 13584 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
W20260807 08:01:13.586058 13703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1253) took 2378 ms (client timeout 2173 ms). Trace:
W20260807 08:01:13.673893 13811 rpcz_store.cc:269] 0807 08:01:11.219234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.559254 (+2340020us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.559385 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.584818 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1265) took 1207 ms. Trace:
I20260807 08:01:13.652016 13587 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:13.590584 16125 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:13.659575 13814 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
W20260807 08:01:13.667686 13809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1145) took 2482 ms (client timeout 2173 ms). Trace:
I20260807 08:01:13.650797 16159 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.544153 16176 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:13.672124 13817 rpcz_store.cc:269] 0807 08:01:11.178170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.556535 (+2378365us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.558887 (+ 2352us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.671838 13585 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
W20260807 08:01:13.673439 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 902527us
I20260807 08:01:13.549840 15973 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:13.486164 16070 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.663358 16167 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:13.651976 16148 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.658666 13589 rpcz_store.cc:276] 0807 08:01:12.362143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.365662 (+ 3519us) service_pool.cc:227] Handling call
0807 08:01:12.764427 (+398765us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:12.771251 (+ 6824us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:12.771289 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:12.771693 (+ 404us) log.cc:842] Serialized 26 byte log entry
0807 08:01:13.419055 (+647362us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:13.419111 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.419190 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:13.535846 (+116656us) raft_consensus.cc:1671] finished
0807 08:01:13.535869 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.537837 (+ 1968us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6452,"raft.queue_time_us":32500,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":80929,"thread_start_us":153209,"threads_started":1,"wal-append.queue_time_us":216477}
W20260807 08:01:13.679260 13809 rpcz_store.cc:269] 0807 08:01:11.182864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.661935 (+2479071us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.665444 (+ 3509us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.683419 16193 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.678458 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.674495 16126 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:13.668922 13807 rpcz_store.cc:276] 0807 08:01:10.492722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.898659 (+405937us) service_pool.cc:227] Handling call
0807 08:01:10.920255 (+ 21596us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:10.920657 (+ 402us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:10.920687 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:10.921187 (+ 500us) log.cc:842] Serialized 26 byte log entry
0807 08:01:13.418171 (+2496984us) raft_consensus.cc:1631] Marking committed up to 1
0807 08:01:13.418272 (+ 101us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.418434 (+ 162us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:13.433955 (+ 15521us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:13.553386 (+119431us) raft_consensus.cc:1671] finished
0807 08:01:13.553408 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.556068 (+ 2660us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":65220,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":162,"spinlock_wait_cycles":127752064,"thread_start_us":2496115,"threads_started":1,"wal-append.queue_time_us":2393260}
W20260807 08:01:13.676966 13703 rpcz_store.cc:269] 0807 08:01:11.207222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:11.207543 (+ 321us) service_pool.cc:227] Handling call
0807 08:01:13.585767 (+2378224us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x7b54003f0554. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:13.585987 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5168693504}
I20260807 08:01:13.561789 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 219) took 2978 ms. Trace:
W20260807 08:01:13.684008 13805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1150) took 2481 ms (client timeout 2169 ms). Trace:
I20260807 08:01:13.685829 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.685690 16053 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:13.686025 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.668810 14622 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:13.673650 14623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 392) took 1294 ms. Trace:
W20260807 08:01:13.684919 13816 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1144) took 2497 ms (client timeout 2248 ms). Trace:
I20260807 08:01:13.691006 14620 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:13.693128 15994 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:13.693542 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:13.697003 13805 rpcz_store.cc:269] 0807 08:01:11.203023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.676723 (+2473700us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.683939 (+ 7216us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260807 08:01:13.675660 13812 rpcz_store.cc:269] 0807 08:01:11.263964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.664379 (+2400415us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.667339 (+ 2960us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.681983 13706 rpcz_store.cc:276] 0807 08:01:12.377494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.702146 (+324652us) service_pool.cc:227] Handling call
0807 08:01:12.703512 (+ 1366us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:13.557698 (+854186us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:13.557763 (+ 65us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.557803 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:13.557832 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.561673 (+ 3841us) spinlock_profiling.cc:243] Waited 854 ms on lock 0x7b5400408b54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:13.561712 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.584614 (+ 22902us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1963391360}
I20260807 08:01:13.678368 16108 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:13.650238 14619 rpcz_store.cc:276] 0807 08:01:12.219442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.557845 (+1338403us) service_pool.cc:227] Handling call
0807 08:01:13.558921 (+ 1076us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:13.559198 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:13.559231 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.559297 (+ 66us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:13.559349 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.559403 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.559542 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.698748 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:13.711623 13590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1552) took 1324 ms. Trace:
W20260807 08:01:13.701272 13816 rpcz_store.cc:269] 0807 08:01:11.181234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.676463 (+2495229us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:13.679013 (+ 2550us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:13.681224 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1555) took 1112 ms. Trace:
I20260807 08:01:13.589663 16195 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.708323 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1554) took 1205 ms. Trace:
I20260807 08:01:13.708240 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.708266 14625 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:13.716704 13590 rpcz_store.cc:276] 0807 08:01:12.383027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.658440 (+1275413us) service_pool.cc:227] Handling call
0807 08:01:13.688399 (+ 29959us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:13.707157 (+ 18758us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:13.707182 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.707216 (+ 34us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:13.707243 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.707310 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.707398 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.709024 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.709174 16098 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.709987 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.688654 13588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1545) took 1204 ms. Trace:
I20260807 08:01:13.710995 13817 raft_consensus.cc:2396] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 08:01:13.714171 16149 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.714547 16176 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.706301 14629 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
W20260807 08:01:13.671911 13804 rpcz_store.cc:269] 0807 08:01:10.487854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.490562 (+ 2708us) service_pool.cc:227] Handling call
0807 08:01:12.719039 (+2228477us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b5800004298. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:13.556511 (+837472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15670656}
I20260807 08:01:13.717841 13591 rpcz_store.cc:276] 0807 08:01:12.501451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.667548 (+1166097us) service_pool.cc:227] Handling call
0807 08:01:13.690418 (+ 22870us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:13.703043 (+ 12625us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:13.703072 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.706606 (+ 3534us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:13.706643 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.706707 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.706812 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.715768 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.722733 13588 rpcz_store.cc:276] 0807 08:01:12.361760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.362640 (+ 880us) service_pool.cc:227] Handling call
0807 08:01:12.466972 (+104332us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:13.511590 (+1044618us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:13.515348 (+ 3758us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.523604 (+ 8256us) raft_consensus.cc:1631] Marking committed up to 1
0807 08:01:13.540717 (+ 17113us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.542977 (+ 2260us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.560662 (+ 17685us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13804,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140}
I20260807 08:01:13.730907 14626 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
W20260807 08:01:13.729624 13817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1153) took 2492 ms (client timeout 2771 ms). Trace:
W20260807 08:01:13.730402 13805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1136) took 2577 ms (client timeout 3050 ms). Trace:
W20260807 08:01:13.732738 13817 rpcz_store.cc:269] 0807 08:01:11.231656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.685583 (+2453927us) service_pool.cc:227] Handling call
0807 08:01:13.724221 (+ 38638us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:13.732833 13805 rpcz_store.cc:269] 0807 08:01:11.153239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.707850 (+2554611us) service_pool.cc:227] Handling call
0807 08:01:13.730308 (+ 22458us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.734078 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.734243 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.734511 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.734818 13817 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:13.735126 13805 raft_consensus.cc:2379] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:13.735595 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 227) took 1723 ms. Trace:
I20260807 08:01:13.735821 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.735877 13805 rpcz_store.cc:276] 0807 08:01:12.012256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.733947 (+1721691us) service_pool.cc:227] Handling call
0807 08:01:13.735546 (+ 1599us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.727376 15911 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.701139 14623 rpcz_store.cc:276] 0807 08:01:12.379126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.283286 (+904160us) service_pool.cc:227] Handling call
0807 08:01:13.672641 (+389355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.735976 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.711018 14622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 393) took 1289 ms. Trace:
W20260807 08:01:13.731580 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 58945.6us
W20260807 08:01:13.737775 13807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 229) took 1706 ms (client timeout 2118 ms). Trace:
W20260807 08:01:13.738083 13811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 228) took 1719 ms (client timeout 1680 ms). Trace:
I20260807 08:01:13.727154 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.713627 16097 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.714745 15976 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.676317 16003 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:13.739509 13809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1532) took 2561 ms (client timeout 2842 ms). Trace:
I20260807 08:01:13.741871 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.734194 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.858701 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:13.865595 13769 connection.cc:232] RPC call timeout handler was delayed by -1.093810518970713s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21610000000000001s for a response
W20260807 08:01:13.864781 13807 rpcz_store.cc:269] 0807 08:01:12.031909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.703576 (+1671667us) service_pool.cc:227] Handling call
0807 08:01:13.737727 (+ 34151us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:13.866601 13809 rpcz_store.cc:269] 0807 08:01:11.178534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.697972 (+2519438us) service_pool.cc:227] Handling call
0807 08:01:13.739461 (+ 41489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.758738 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:13.866185 13811 rpcz_store.cc:269] 0807 08:01:12.019000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.696689 (+1677689us) service_pool.cc:227] Handling call
0807 08:01:13.738033 (+ 41344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.832414 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:13.760490 13814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 218) took 3263 ms (client timeout 1900 ms). Trace:
I20260807 08:01:13.765852 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.835017 14583 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:13.872373 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.873099 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:13.874003 13700 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
W20260807 08:01:13.874146 13816 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1138) took 2720 ms (client timeout 3288 ms). Trace:
I20260807 08:01:13.737110 16122 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.882288 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.812887 13708 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:13.882670 13817 raft_consensus.cc:2396] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 08:01:13.736585 14619 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:01:13.874217 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.887507 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.761063 16085 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:13.884246 13805 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
W20260807 08:01:13.885504 13816 rpcz_store.cc:269] 0807 08:01:11.155581 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.718001 (+2562420us) service_pool.cc:227] Handling call
0807 08:01:13.874070 (+156069us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.921444 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:13.899158 13809 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
W20260807 08:01:13.871374 13814 rpcz_store.cc:269] 0807 08:01:10.539471 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.629428 (+ 89957us) service_pool.cc:227] Handling call
0807 08:01:11.073387 (+443959us) spinlock_profiling.cc:243] Waited 76.8 ms on lock 0x7b5800004298. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:13.760254 (+2686867us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b54001fc2d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:13.760399 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6073364224}
I20260807 08:01:13.900528 13589 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
W20260807 08:01:13.909034 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 139601us
I20260807 08:01:13.890746 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.762986 15973 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.800304 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.740020 13661 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:13.929682 14583 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:13.793412 16053 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.930586 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.932574 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.933322 13816 raft_consensus.cc:2396] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:13.934958 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.935261 14629 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:13.882817 16198 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.742915 14622 rpcz_store.cc:276] 0807 08:01:12.414536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.500613 (+1086077us) service_pool.cc:227] Handling call
0807 08:01:13.703327 (+202714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:13.799288 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.762552 14620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 387) took 1614 ms. Trace:
I20260807 08:01:13.759776 16147 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.944802 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.940572 13590 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:13.800259 16167 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.943586 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.806443 16105 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:13.953064 15994 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:13.731510 16108 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:13.930729 16176 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:13.934939 14627 raft_consensus.cc:2354] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:13.899659 16121 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.952473 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.944407 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.953858 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.931282 16172 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:13.797422 16195 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.955549 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:13.955331 15994 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:13.955251 13580 raft_consensus.cc:2379] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:13.939881 13588 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:13.954578 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:13.968990 13540 connection.cc:232] RPC call timeout handler was delayed by -1.3049520521890372s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2167s for a response
I20260807 08:01:13.967625 13814 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
W20260807 08:01:13.973873 13540 connection.cc:232] RPC call timeout handler was delayed by -1.3049520521890372s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2167s for a response
I20260807 08:01:13.971015 15994 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:13.720117 13579 rpcz_store.cc:276] 0807 08:01:12.569330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.674076 (+1104746us) service_pool.cc:227] Handling call
0807 08:01:13.680246 (+ 6170us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:13.680749 (+ 503us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:13.680786 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.680848 (+ 62us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:13.680913 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.680983 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.681125 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4118,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":8024}
I20260807 08:01:13.971484 13817 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:13.973219 13809 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:13.966933 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:13.954947 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:13.964851 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:13.961943 16176 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:13.969403 13812 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
W20260807 08:01:13.974203 13540 connection.cc:232] RPC call timeout handler was delayed by -1.2608520521316677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22160000000000002s for a response
I20260807 08:01:13.972040 14583 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:13.954909 15891 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:13.997260 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.045760 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.007786 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:13.955993 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.008873 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:14.016849 13540 connection.cc:232] RPC call timeout handler was delayed by -1.2608520521316677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22160000000000002s for a response
W20260807 08:01:14.048628 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 59690.6us
I20260807 08:01:13.954347 14623 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:13.996737 14629 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:14.003418 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:13.988360 13805 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:13.971300 16003 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.008167 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.054697 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:13.991629 15976 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:14.057858 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.059090 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.061564 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:13.952139 14620 rpcz_store.cc:276] 0807 08:01:12.148456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.536086 (+1387630us) service_pool.cc:227] Handling call
0807 08:01:13.762273 (+226187us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b5400680454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:13.762459 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54065024}
I20260807 08:01:14.063241 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.049659 16176 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:14.064272 13540 connection.cc:232] RPC call timeout handler was delayed by -1.0076136679854244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16240000000000002s for a response
I20260807 08:01:14.065225 16167 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.065584 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.067025 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.068775 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.009368 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.070013 16195 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 08:01:14.070907 13580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1543) took 1719 ms (client timeout 1918 ms). Trace:
W20260807 08:01:14.058835 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 331192us
W20260807 08:01:13.994063 14618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 561) took 3562 ms (client timeout 1582 ms). Trace:
I20260807 08:01:13.990834 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.074921 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:14.063256 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 99588.7us
I20260807 08:01:14.076431 13711 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:14.076362 13709 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:14.050952 13804 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:14.077028 13816 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
W20260807 08:01:14.057137 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 226) took 2067 ms (client timeout 2286 ms). Trace:
I20260807 08:01:14.083431 14583 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.082759 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.019047 16139 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.056886 14622 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:14.085203 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.081172 14629 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:01:14.077792 16195 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:14.092522 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.100862 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.092794 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.084710 13701 raft_consensus.cc:2354] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:14.004529 16154 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.098915 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:13.992779 16144 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.091804 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.077111 16085 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:13.699088 13808 rpcz_store.cc:276] 0807 08:01:10.539880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:11.086444 (+546564us) service_pool.cc:227] Handling call
0807 08:01:11.132264 (+ 45820us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:11.147902 (+ 15638us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 08:01:12.869766 (+1721864us) log.cc:842] Serialized 14 byte log entry
0807 08:01:13.447920 (+578154us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:13.466395 (+ 18475us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:13.487616 (+ 21221us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:13.508953 (+ 21337us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:13.518481 (+ 9528us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":968138,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":46889,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":2453,"thread_start_us":1714714,"threads_started":2,"wal-append.queue_time_us":209326}
I20260807 08:01:14.099184 13817 raft_consensus.cc:2396] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 08:01:14.106810 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:14.102967 16195 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.111215 13703 raft_consensus.cc:2354] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:14.108738 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1134) took 2951 ms. Trace:
I20260807 08:01:14.112969 13814 rpcz_store.cc:276] 0807 08:01:11.152554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.081090 (+2928536us) service_pool.cc:227] Handling call
0807 08:01:14.098993 (+ 17903us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.099159 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.099184 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.103249 (+ 4065us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:14.103288 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.103345 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.103466 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.115072 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:14.116716 14626 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.117321 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.085886 14625 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:14.102044 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.110556 13705 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
W20260807 08:01:14.079159 13580 rpcz_store.cc:269] 0807 08:01:12.351376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:12.353573 (+ 2197us) service_pool.cc:227] Handling call
0807 08:01:13.296838 (+943265us) spinlock_profiling.cc:243] Waited 4.85 ms on lock 0x7b5800001898. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:14.070696 (+773858us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b54001e3a54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:14.070851 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1015944448}
W20260807 08:01:14.085879 13812 rpcz_store.cc:269] 0807 08:01:11.984716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:13.726821 (+1742105us) service_pool.cc:227] Handling call
0807 08:01:14.048295 (+321474us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.106634 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:14.127388 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.126405 13710 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:14.105366 14627 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:14.135178 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.135777 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.129328 15976 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:14.139227 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.167636 13540 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:01:14.080745 14618 rpcz_store.cc:269] 0807 08:01:10.431331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.438997 (+ 7666us) service_pool.cc:227] Handling call
0807 08:01:12.032301 (+1593304us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00bbce 000000000034aea9 00007f6b21e37a30 00007f6b21efd7b4 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:13.993638 (+1961337us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54003309d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:13.993841 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":545651456}
I20260807 08:01:14.173249 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.174441 13709 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:14.136921 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1148) took 2936 ms. Trace:
I20260807 08:01:14.219321 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1155) took 1987 ms. Trace:
I20260807 08:01:14.206881 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:14.229761 13814 rpcz_store.cc:276] 0807 08:01:12.231548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.124769 (+1893221us) service_pool.cc:227] Handling call
0807 08:01:14.133550 (+ 8781us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.218859 (+ 85309us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.218899 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.218981 (+ 82us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:14.219023 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.219101 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.219254 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.138963 14630 raft_consensus.cc:2354] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4.
I20260807 08:01:14.137781 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1143) took 2953 ms. Trace:
W20260807 08:01:14.184087 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 55896.2us
I20260807 08:01:14.137979 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 224) took 3527 ms. Trace:
I20260807 08:01:14.232625 13811 rpcz_store.cc:276] 0807 08:01:11.180951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.081087 (+2900136us) service_pool.cc:227] Handling call
0807 08:01:14.110870 (+ 29783us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.129907 (+ 19037us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.129944 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.133665 (+ 3721us) raft_consensus.cc:1631] Marking committed up to 1
0807 08:01:14.133717 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.133794 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.133922 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.233560 14583 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.233795 13805 rpcz_store.cc:276] 0807 08:01:10.606997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.058414 (+3451417us) service_pool.cc:227] Handling call
0807 08:01:14.094413 (+ 35999us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.130058 (+ 35645us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.130099 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.134056 (+ 3957us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:14.134105 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.134176 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.134326 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.231769 13708 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.182403 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.229580 13816 rpcz_store.cc:276] 0807 08:01:11.196242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.092188 (+2895946us) service_pool.cc:227] Handling call
0807 08:01:14.119436 (+ 27248us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.129082 (+ 9646us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.129131 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.132318 (+ 3187us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:14.132373 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.132451 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.132614 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.204181 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1154) took 2116 ms. Trace:
I20260807 08:01:14.221170 14626 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
W20260807 08:01:14.237830 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 103879us
I20260807 08:01:14.237919 13817 rpcz_store.cc:276] 0807 08:01:12.088733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.114100 (+2025367us) service_pool.cc:227] Handling call
0807 08:01:14.134436 (+ 20336us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.165896 (+ 31460us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.165942 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.165987 (+ 45us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:14.203737 (+ 37750us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.203857 (+ 120us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.204067 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.169421 14623 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:14.241029 16154 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:14.242812 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.221143 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1137) took 3058 ms. Trace:
I20260807 08:01:14.244307 13809 rpcz_store.cc:276] 0807 08:01:11.153598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.082650 (+2929052us) service_pool.cc:227] Handling call
0807 08:01:14.116641 (+ 33991us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.204151 (+ 87510us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.204182 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.204223 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:14.211725 (+ 7502us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.211806 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.211916 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.175578 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.169144 16139 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.132735 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:14.137749 16108 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:14.212371 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.250538 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.234776 14622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 390) took 1863 ms. Trace:
I20260807 08:01:14.251997 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.250687 13814 raft_consensus.cc:1261] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:14.238272 15976 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.279397 15976 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.284024 14619 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:14.283763 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.284705 13710 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:14.285861 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.286638 13710 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:14.287742 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.288552 13710 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:14.277791 14622 rpcz_store.cc:276] 0807 08:01:12.353955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.119309 (+1765354us) service_pool.cc:227] Handling call
0807 08:01:14.205827 (+ 86518us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.206109 (+ 282us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.206148 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.206197 (+ 49us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:14.206240 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.206311 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.216840 (+ 10529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.149837 13700 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:14.278344 13537 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.278764 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.295015 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:14.295615 13711 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.280071 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.291774 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.239189 16085 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:14.292707 13708 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:14.295943 13769 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:14.295261 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 46785.6us
I20260807 08:01:14.296407 13705 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:14.299408 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:14.301381 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.297273 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:14.302857 13705 raft_consensus.cc:2379] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:14.302971 13709 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.298089 13661 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:14.304517 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 72488.8us
I20260807 08:01:14.306274 13537 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.309154 13710 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:14.309767 13807 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:14.250311 13804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1149) took 3048 ms. Trace:
I20260807 08:01:14.293821 16105 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.288044 15976 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:14.311839 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 58622.3us
I20260807 08:01:14.325178 13709 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:14.298372 16085 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:14.310873 16130 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:14.321667 16121 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.343113 16121 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:14.343581 16121 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.322067 16147 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.314404 13540 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.311645 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.294425 13701 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:14.308235 16144 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:14.349974 14630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 383) took 3886 ms (client timeout 2167 ms). Trace:
W20260807 08:01:14.350454 14630 rpcz_store.cc:269] 0807 08:01:10.463710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:10.466851 (+ 3141us) service_pool.cc:227] Handling call
0807 08:01:11.542268 (+1075417us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:14.349763 (+2807495us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b54002b07d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:14.349909 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4381955840}
I20260807 08:01:14.313459 16105 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.310480 14620 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:14.341960 16085 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.344003 15976 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:14.357756 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 56097.6us
I20260807 08:01:14.347314 16147 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.362499 13769 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.354673 16105 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.367465 13540 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:01:14.378280 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 54612.2us
I20260807 08:01:14.329819 16167 consensus_queue.cc:1035] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260807 08:01:14.367684 16147 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.355837 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1156) took 2122 ms. Trace:
I20260807 08:01:14.408665 16126 raft_consensus.cc:2781] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.311045 13804 rpcz_store.cc:276] 0807 08:01:11.200701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.105314 (+2904613us) service_pool.cc:227] Handling call
0807 08:01:14.122632 (+ 17318us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.206543 (+ 83911us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.211469 (+ 4926us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.233011 (+ 21542us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:14.236813 (+ 3802us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.241698 (+ 4885us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.248819 (+ 7121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.358970 16172 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:14.404526 14629 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:14.361474 16144 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:14.409960 13769 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.434084 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.435209 13814 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:14.410060 16126 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.412961 16097 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.414160 16172 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:14.440210 16172 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.440411 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.441818 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.443169 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.443758 13812 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:14.444037 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.444837 14620 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:14.444653 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.445756 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.445863 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.438715 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.446545 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.447897 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.417776 16144 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:14.445559 16176 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.451179 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.453879 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.454545 13537 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.458266 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.439558 16126 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.455295 16015 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.456115 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.476320 13816 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:14.439700 16097 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:14.476955 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.478724 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.481074 13805 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.483950 13814 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:14.484992 13809 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:14.476155 13811 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:14.450647 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.488998 13661 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:14.489984 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.453953 16144 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.493275 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.493669 13817 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:14.480726 16176 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:14.494928 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 45461.4us
I20260807 08:01:14.498899 13816 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:14.496254 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.482623 16015 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.485778 16097 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.409394 13808 rpcz_store.cc:276] 0807 08:01:12.233800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.121621 (+1887821us) service_pool.cc:227] Handling call
0807 08:01:14.233338 (+111717us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.322852 (+ 89514us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.325176 (+ 2324us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:14.329027 (+ 3851us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:14.344208 (+ 15181us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.347476 (+ 3268us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.354586 (+ 7110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:14.505265 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.499586 16130 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:14.483073 16195 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:14.507982 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 273837us
I20260807 08:01:14.511404 13537 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.491825 14620 raft_consensus.cc:2362] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3. Re-sending same reply.
I20260807 08:01:14.502521 16015 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.503307 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.509925 16130 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:14.531368 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 314275us
I20260807 08:01:14.532595 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.535333 14583 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:14.524547 16130 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.536690 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.540077 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.528707 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.542821 16176 raft_consensus.cc:2726] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:14.545290 16172 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.547273 16172 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.547822 16172 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:14.548171 16172 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:14.548470 16172 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.529101 13812 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:14.549351 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.550259 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.550694 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.550825 13812 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.551244 13809 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.541602 14625 raft_consensus.cc:1226] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260807 08:01:14.566282 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.570251 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.570967 13817 raft_consensus.cc:2354] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:14.571301 16085 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.571712 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.572836 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.573210 16085 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:14.575641 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 211284us
I20260807 08:01:14.578723 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.579216 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.580670 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.581593 13805 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:14.551065 15994 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.586009 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.588028 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.593463 13711 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:14.599015 14580 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:14.585672 16085 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:14.611678 14580 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.589910 15976 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.613106 13540 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:14.613181 14618 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:14.616501 15976 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.621503 15976 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.614387 16198 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 08:01:14.634222 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 122957us
I20260807 08:01:14.639076 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:14.639631 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.614805 16085 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.638937 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:14.641793 13589 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.619237 16195 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.642067 14580 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:14.643512 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:14.643563 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:14.644124 13587 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.646001 13585 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.648396 13592 raft_consensus.cc:2379] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:14.647787 13537 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:14.643808 16085 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:14.647873 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 112833us
I20260807 08:01:14.662065 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:14.663995 16226 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:14.664426 16226 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:14.664824 16226 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.665972 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.666833 13805 raft_consensus.cc:2354] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:14.667234 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.668568 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:14.669687 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 56672.5us
I20260807 08:01:14.667618 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.670456 13540 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:14.666857 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:14.665867 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:14.687604 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.688405 14626 raft_consensus.cc:2379] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:14.688634 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 569) took 1145 ms. Trace:
I20260807 08:01:14.689056 14625 rpcz_store.cc:276] 0807 08:01:13.568372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.138010 (+569638us) service_pool.cc:227] Handling call
0807 08:01:14.243518 (+105508us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:14.579930 (+336412us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.688352 (+108422us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54006ac2d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:14.688558 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":627235072}
I20260807 08:01:14.680294 16195 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:14.680791 16216 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:14.692461 16216 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:14.692863 16216 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.692929 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.681828 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 572) took 1091 ms. Trace:
I20260807 08:01:14.693714 13588 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:14.648505 16015 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.697914 13814 raft_consensus.cc:2379] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 2: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:14.698096 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.698541 15994 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.683032 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.616854 16121 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.701022 16121 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:14.701593 13816 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:14.701392 16226 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.703091 16221 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:14.693676 14627 rpcz_store.cc:276] 0807 08:01:13.590678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.165596 (+574918us) service_pool.cc:227] Handling call
0807 08:01:14.234385 (+ 68789us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:14.240753 (+ 6368us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:14.240807 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:14.241381 (+ 574us) log.cc:842] Serialized 26 byte log entry
0807 08:01:14.537862 (+296481us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:14.539429 (+ 1567us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.551597 (+ 12168us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:14.681600 (+130003us) raft_consensus.cc:1671] finished
0807 08:01:14.681630 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:14.681769 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2415,"raft.queue_time_us":924,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":14459,"thread_start_us":216560,"threads_started":1,"wal-append.queue_time_us":376504}
I20260807 08:01:14.703495 16226 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.703764 16221 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.635677 16097 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.690588 14618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 578) took 1059 ms. Trace:
I20260807 08:01:14.690925 14622 raft_consensus.cc:1226] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:14.698761 13580 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:14.684823 13812 raft_consensus.cc:2396] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260807 08:01:14.704330 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 62343.2us
I20260807 08:01:14.669416 13817 raft_consensus.cc:2379] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:14.722361 13585 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:14.717615 16221 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:14.726748 16224 raft_consensus.cc:2741] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 08:01:14.727005 16221 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.698815 16015 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:14.728693 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.730959 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.731321 16147 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.732311 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.732162 16172 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.733021 13591 raft_consensus.cc:2379] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:14.733742 16130 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.734575 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.717873 14618 rpcz_store.cc:276] 0807 08:01:13.631028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:14.283255 (+652227us) service_pool.cc:227] Handling call
0807 08:01:14.283600 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:14.620976 (+337376us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:14.690342 (+ 69366us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b54005235d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:14.690532 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":727530752}
I20260807 08:01:14.735847 16130 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.735903 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.736348 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.737030 16172 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.737135 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:14.645650 16215 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:14.738106 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.742697 14626 raft_consensus.cc:2354] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:14.739231 14583 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:14.734210 16097 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.743950 13809 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:14.762642 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.740183 13587 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
W20260807 08:01:14.744597 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 74685.6us
I20260807 08:01:14.745083 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.745829 13580 raft_consensus.cc:1226] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
W20260807 08:01:14.741384 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 71015.6us
I20260807 08:01:14.747264 16126 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.748220 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.779057 14626 raft_consensus.cc:2354] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:14.779433 14629 raft_consensus.cc:2379] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:14.748685 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.749003 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.792008 14583 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.756567 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.747330 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.793999 13591 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:14.794433 13585 raft_consensus.cc:2354] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:14.783730 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:14.780864 13589 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:14.733404 16147 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.754460 16144 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.762902 15994 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:14.786984 16085 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.788693 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.797775 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:14.798056 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:14.798695 14629 raft_consensus.cc:2354] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:14.797791 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:14.789319 16176 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:14.801219 14626 raft_consensus.cc:2379] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:14.752769 16105 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:14.783643 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 135911us
I20260807 08:01:14.748226 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:14.781757 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 57022.2us
I20260807 08:01:14.802486 14580 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.803408 14580 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.804224 13592 raft_consensus.cc:2379] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:14.804282 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.749877 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.805794 16195 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.794880 13590 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.806231 16195 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.806649 16195 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.797847 16193 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:14.807508 16193 raft_consensus.cc:686] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:14.807644 16198 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:14.808092 16198 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:14.808470 16198 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.808495 16193 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.802480 13537 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.796017 16130 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:14.796377 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.814348 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:14.749344 15976 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.800515 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:14.801414 13805 raft_consensus.cc:1226] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:14.815510 13587 raft_consensus.cc:2379] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:14.734222 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.824097 16130 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.825475 16122 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:14.825920 16122 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:14.826283 16122 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.826727 16130 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.828114 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.828830 13700 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:14.830673 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.831209 16085 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.831514 13702 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:14.832144 13580 raft_consensus.cc:1226] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260807 08:01:14.833079 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.833565 16144 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.833956 13700 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:14.834540 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:14.835011 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.805389 16126 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:14.835485 13589 raft_consensus.cc:2354] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:14.802623 14583 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.821254 13590 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:14.793995 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:14.838102 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.758658 16215 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.751582 13579 raft_consensus.cc:1261] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:14.838800 13710 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:14.755627 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:14.838701 16144 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.839944 16144 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.840497 16144 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.840274 16126 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:14.840582 16215 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:14.841156 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 47243.3us
I20260807 08:01:14.803239 16176 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.841753 16176 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.835759 13702 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:14.839887 14618 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:14.819000 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.844594 13769 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.845836 13711 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:14.820569 16167 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:14.846793 16167 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.846958 16221 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.822635 14630 raft_consensus.cc:1226] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:14.822158 14620 raft_consensus.cc:1226] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:14.805246 14580 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.816732 15976 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:14.847399 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:14.816705 16198 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.848966 16167 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.850457 14622 raft_consensus.cc:2354] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:14.851411 16198 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.851344 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.852315 13711 raft_consensus.cc:2379] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:14.824096 16105 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:14.854439 13540 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.852722 16176 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.814348 13537 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:14.856259 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 53708.4us
I20260807 08:01:14.911196 14583 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.912256 15976 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:14.912858 15976 raft_consensus.cc:686] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:14.914101 15976 consensus_queue.cc:237] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.864952 16195 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.869633 16105 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.917246 16105 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.917714 16105 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.918520 16195 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.921054 16237 raft_consensus.cc:2781] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.921696 16237 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.922168 16237 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.871516 13540 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:01:14.924019 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 52608.1us
I20260807 08:01:14.925012 13540 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:14.925840 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:14.926573 16172 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:14.926931 16172 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:14.927217 16224 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.927709 16224 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.883162 16015 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.928004 16236 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:14.928516 16015 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.928632 16236 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.871968 16085 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.929083 16015 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.929137 16236 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.929309 16085 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.930061 16085 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:01:14.931463 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 322284us
I20260807 08:01:14.846855 16226 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:14.887580 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.886612 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.932746 16226 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:14.888384 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.933204 16226 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.933509 13702 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:14.889427 16221 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.892302 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.934718 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.935426 13702 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:14.935726 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.933156 13700 raft_consensus.cc:2379] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:14.934561 13540 leader_election.cc:398] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4.
I20260807 08:01:14.936404 13710 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:14.936734 13540 leader_election.cc:401] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:14.895129 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.933588 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.937786 13701 raft_consensus.cc:2379] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:14.896950 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.897341 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.938730 13657 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:14.895478 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:14.939509 16230 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:14.939867 13703 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:14.939985 16230 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:14.897774 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:14.940429 16230 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:14.940928 13708 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:14.942366 16147 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4.
I20260807 08:01:14.943343 16167 consensus_queue.cc:1035] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:14.944108 13657 leader_election.cc:398] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:14.944394 13657 leader_election.cc:401] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 08:01:14.943336 13540 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:14.946313 16167 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:14.948622 13657 leader_election.cc:398] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:14.948972 13657 leader_election.cc:401] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260807 08:01:14.907783 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 60489.7us
I20260807 08:01:14.950865 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:14.952019 16234 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.952529 16234 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:14.953064 16234 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.908663 16144 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.869221 16176 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:14.927317 16172 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:14.928067 16224 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.902737 16122 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.893746 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:14.959657 13705 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:14.837221 16216 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.961397 14627 raft_consensus.cc:2379] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:14.870546 15994 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:14.962670 13428 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56598 (request call id 299) took 5589 ms. Trace:
I20260807 08:01:14.963120 15994 raft_consensus.cc:686] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:14.963572 16216 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:14.964193 15994 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:14.963016 13428 rpcz_store.cc:276] 0807 08:01:09.373220 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:09.373377 (+ 157us) service_pool.cc:227] Handling call
0807 08:01:09.823241 (+449864us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b54000b0c54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0af0 00007f6b1cf73f2a 00007f6b22ec5d6c 00007f6b22fbfc3e 00007f6b212d1fb4 00007f6b212d1f42 00007f6b212d1e71 00007f6b212d1dec 00007f6b212d10a2 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:14.962612 (+5139371us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:13.699547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:13.953495 (+253948us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:13.961385 (+ 7890us) write_op.cc:435] Acquired schema lock
0807 08:01:13.964584 (+ 3199us) tablet.cc:660] Decoding operations
0807 08:01:14.005568 (+ 40984us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:14.005692 (+ 124us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:14.005752 (+ 60us) tablet.cc:683] Acquiring locks for 1 operations
0807 08:01:14.006078 (+ 326us) tablet.cc:699] Row locks acquired
0807 08:01:14.006133 (+ 55us) write_op.cc:260] PREPARE: finished
0807 08:01:14.006309 (+ 176us) write_op.cc:270] Start()
0807 08:01:14.006482 (+ 173us) write_op.cc:276] Timestamp: P: 1786089674006237 usec, L: 0
0807 08:01:14.006526 (+ 44us) op_driver.cc:331] REPLICATION: starting
0807 08:01:14.086067 (+ 79541us) log.cc:842] Serialized 536 byte log entry
0807 08:01:14.653315 (+567248us) op_driver.cc:444] REPLICATION: finished
0807 08:01:14.952976 (+299661us) write_op.cc:301] APPLY: starting
0807 08:01:14.953392 (+ 416us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:14.953543 (+ 151us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:14.953738 (+ 195us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:14.954113 (+ 375us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:14.954752 (+ 639us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0807 08:01:14.954821 (+ 69us) write_op.cc:312] APPLY: finished
0807 08:01:14.955134 (+ 313us) log.cc:842] Serialized 23 byte log entry
0807 08:01:14.957432 (+ 2298us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:14.957532 (+ 100us) write_op.cc:454] Released schema lock
0807 08:01:14.957610 (+ 78us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:14.958012 (+ 402us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:14.958057 (+ 45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2129105,"reported_tablets":1,"spinlock_wait_cycles":738521472,"child_traces":[["op",{"apply.queue_time_us":299321,"apply.run_cpu_time_us":2725,"apply.run_wall_time_us":5393,"num_ops":1,"prepare.queue_time_us":170849,"prepare.run_cpu_time_us":181991,"prepare.run_wall_time_us":727624,"raft.queue_time_us":108779,"raft.run_cpu_time_us":43019,"raft.run_wall_time_us":257382,"replication_time_us":646674,"thread_start_us":703123,"threads_started":4,"wal-append.queue_time_us":419743}]]}
W20260807 08:01:14.938539 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 124383us
I20260807 08:01:14.976946 16144 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:14.979224 16167 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:14.983724 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:14.983649 16167 consensus_queue.cc:1046] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:14.984877 16130 raft_consensus.cc:3032] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:14.984979 16147 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:14.985332 16130 raft_consensus.cc:729] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:14.985469 16147 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:14.985944 16147 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.986373 16130 consensus_queue.cc:260] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:14.987797 16130 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:14.991987 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:14.999763 16122 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:15.000414 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 195221us
I20260807 08:01:15.001261 14580 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:15.010097 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 59318.4us
I20260807 08:01:15.003839 13537 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.022712 16167 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:15.023779 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:15.023903 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.025758 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:15.031559 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.033036 13705 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:15.034626 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.034730 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:15.035456 13708 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:15.035903 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:15.036509 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.036722 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.023315 16167 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.037878 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.038173 16105 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.039348 16167 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.044530 13808 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:15.048231 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:15.049265 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:15.027189 14580 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.028031 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:15.028404 13816 raft_consensus.cc:1226] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.060137 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.061182 13817 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:15.062829 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:15.063776 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:15.063318 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 59563.1us
I20260807 08:01:15.064771 13537 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.063256 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.071388 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:15.070418 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 50979.3us
I20260807 08:01:15.076938 14580 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.081024 16122 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:15.083037 16105 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:15.051877 16243 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:15.084298 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.085208 13703 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:15.085229 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.091969 16236 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.082877 16122 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:15.094939 16122 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:15.094601 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.099830 13589 raft_consensus.cc:1226] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:15.100260 16226 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.102777 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.125452 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.126377 13590 raft_consensus.cc:2354] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
W20260807 08:01:15.128319 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 52947.6us
W20260807 08:01:15.130712 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 53881.5us
I20260807 08:01:15.084246 16243 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:15.139472 13537 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:15.145411 16226 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.148124 16195 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:15.149029 16193 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:15.059065 16231 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:15.070075 16144 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:15.162047 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.030509 16015 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.184722 13703 raft_consensus.cc:2379] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:15.073361 16147 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.126999 16097 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:15.125334 15976 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:15.170579 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.181254 16130 consensus_queue.cc:1046] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260807 08:01:15.182883 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 43539.2us
I20260807 08:01:15.187383 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.160197 16105 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:15.079734 16237 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.163582 16243 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.187305 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.211072 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:15.195385 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 169703us
I20260807 08:01:15.195760 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.212072 13584 raft_consensus.cc:1226] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.212105 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.208781 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.208209 13585 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260807 08:01:15.211094 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.193595 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.212420 13816 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.209553 13708 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.183701 16144 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:15.215878 13769 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.182873 16259 raft_consensus.cc:2726] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:15.214751 14580 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.230680 13816 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:15.192339 13812 raft_consensus.cc:2354] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:15.217144 16236 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.217727 16226 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:15.232012 16226 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:15.232479 16226 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:15.213824 16235 log.cc:924] Time spent T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20260807 08:01:15.193151 16193 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.189852 16097 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:15.218456 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.235596 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.236137 16193 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:15.236312 13701 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:15.236454 16015 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.237151 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.237846 16224 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.238844 16097 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.239174 13585 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.228487 16237 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:15.228421 16085 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.218935 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.229496 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.221069 16144 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.240697 16224 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.230216 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.244469 13587 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:15.222383 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.245234 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.222762 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.245850 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.246353 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.246410 13702 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.247568 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.248574 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.217064 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.249100 13587 raft_consensus.cc:2454] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:15.221267 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.248914 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.250838 13584 raft_consensus.cc:1226] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:15.251607 16231 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:15.252029 16231 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.224206 16254 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:15.238323 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.243259 13591 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:15.252962 16254 raft_consensus.cc:686] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:15.224736 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.254118 13706 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.224835 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.254616 16254 consensus_queue.cc:237] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.255617 13591 raft_consensus.cc:1226] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.225186 13580 raft_consensus.cc:1226] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:15.224439 16105 raft_consensus.cc:686] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:15.225544 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.257153 16105 consensus_queue.cc:237] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.194437 15976 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:15.224041 16230 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.190047 16195 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.226495 13769 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.219349 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.252157 13584 raft_consensus.cc:1226] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.226670 16234 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.276089 14630 raft_consensus.cc:1226] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.277182 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:15.242592 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:15.291960 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 83197.9us
I20260807 08:01:15.291846 13705 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:15.243819 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.293576 16085 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.294781 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.222383 16147 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.297454 13705 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.298772 16230 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.299469 13709 raft_consensus.cc:2379] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:15.300968 14622 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:15.301653 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:15.295944 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:15.302373 14620 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:15.302486 16195 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:15.302614 14626 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:01:15.303072 13702 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:15.303237 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:15.226831 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.307060 13705 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:15.296255 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.296741 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.309098 16216 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:15.286243 15976 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.314963 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.319849 13428 catalog_manager.cc:5495] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 1 to 2, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 2 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:15.321854 16222 raft_consensus.cc:2741] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260807 08:01:15.226864 16172 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.327461 16172 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.330071 13708 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
W20260807 08:01:15.321762 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 95341.8us
I20260807 08:01:15.336097 16255 consensus_queue.cc:1046] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:15.335037 13428 catalog_manager.cc:5495] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:15.339612 16126 consensus_queue.cc:1046] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:15.337695 16255 raft_consensus.cc:3032] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
W20260807 08:01:15.340698 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 156743us
I20260807 08:01:15.341006 13428 catalog_manager.cc:5495] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } }
I20260807 08:01:15.349243 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.347532 13428 catalog_manager.cc:5495] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 2 to 3, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 3 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:15.352263 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.347568 13706 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:15.357599 13805 raft_consensus.cc:2379] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:15.358589 13708 raft_consensus.cc:2354] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:15.358500 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:15.359529 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.360100 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.360879 13711 raft_consensus.cc:2354] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:15.360788 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.361534 13710 raft_consensus.cc:2379] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:15.360162 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.361871 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.362527 13703 raft_consensus.cc:2379] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:15.362761 13700 raft_consensus.cc:2354] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:15.363600 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:15.363883 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.364444 14619 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:15.364621 16015 raft_consensus.cc:3032] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:15.364836 13702 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:15.364980 16015 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:15.346248 16255 raft_consensus.cc:729] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:15.363610 16097 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.359542 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.366884 13812 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.366520 16255 consensus_queue.cc:260] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.367893 16255 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.368395 16097 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:15.367017 13589 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:15.369467 13428 catalog_manager.cc:5495] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 3 to 4, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 4 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:15.370535 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:15.370796 16226 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.370857 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.374264 13428 catalog_manager.cc:5495] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: config changed from index 2 to 3, 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:15.380030 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.382309 16144 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.382881 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.383353 16226 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.383582 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.384464 13812 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:15.384402 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.385196 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.385803 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.411075 13708 raft_consensus.cc:2379] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:15.411070 14626 raft_consensus.cc:2379] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:15.386899 16167 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:15.386175 16015 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.388397 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.391532 13657 leader_election.cc:398] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:15.399735 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.424486 13709 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
W20260807 08:01:15.400673 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 51136us
I20260807 08:01:15.400805 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:15.425557 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 165993us
I20260807 08:01:15.409015 13706 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
W20260807 08:01:15.412611 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 197936us
I20260807 08:01:15.425101 13657 leader_election.cc:401] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:15.425995 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.392005 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.391901 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.392405 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.435491 14630 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:15.387933 16234 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.427026 13537 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.436126 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.409644 13702 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.343775 16231 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.345643 15976 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.456414 13579 raft_consensus.cc:2379] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:15.413848 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.465638 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:15.457490 16144 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:15.461315 13661 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.462957 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:15.464352 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 72961.2us
I20260807 08:01:15.471196 13537 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.438277 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.424434 16167 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.467679 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:15.475572 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.472920 16122 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.476547 14622 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:15.477764 14627 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:15.477916 16122 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:15.480592 14625 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260807 08:01:15.482600 16167 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:15.484493 16279 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:15.483270 16281 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:15.488859 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.492877 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.470643 13701 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.497368 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.497975 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.500772 13805 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.501295 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.472255 14620 raft_consensus.cc:2379] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 2: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:15.503334 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.511360 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.472060 16269 raft_consensus.cc:2741] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260807 08:01:15.488844 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:15.519268 13769 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:15.503361 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.512050 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.503870 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.533653 16226 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:15.535454 16231 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:15.538385 15976 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:15.538933 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.540359 13808 raft_consensus.cc:1226] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:15.499441 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:15.541495 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.541802 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.549724 13804 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.549994 16139 raft_consensus.cc:3032] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:15.550343 16139 raft_consensus.cc:729] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:15.551182 16139 consensus_queue.cc:260] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.553129 13769 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:15.554030 16253 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:15.554365 13769 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:15.555137 16253 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:15.555626 16015 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:15.517127 13814 raft_consensus.cc:2379] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 2: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:15.514310 14619 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:15.514348 13657 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:15.562089 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 47832.6us
I20260807 08:01:15.564443 13805 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
W20260807 08:01:15.517722 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 75773.3us
I20260807 08:01:15.566105 14583 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:15.496855 16147 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:15.587792 16147 raft_consensus.cc:686] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:15.585794 16139 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:15.502724 16243 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.598309 14626 raft_consensus.cc:2379] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:15.599611 16144 consensus_queue.cc:1046] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:15.601073 16085 raft_consensus.cc:3032] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:15.601543 16085 raft_consensus.cc:729] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:15.603796 16085 consensus_queue.cc:260] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.607105 13537 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.586443 16105 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:15.586621 13701 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:15.587018 16053 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:15.505847 14580 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.555908 13769 leader_election.cc:398] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:15.611687 13769 leader_election.cc:401] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:15.588706 13814 raft_consensus.cc:1226] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.547602 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.547935 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.543874 14630 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:15.545922 14623 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:15.590955 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:15.592268 13817 raft_consensus.cc:1226] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.605468 16085 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.592916 13807 raft_consensus.cc:2396] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260807 08:01:15.615978 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 60191.1us
I20260807 08:01:15.596724 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:15.595474 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:15.607549 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 146321us
I20260807 08:01:15.586031 13804 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:15.620421 16285 consensus_queue.cc:1046] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:15.621347 14622 raft_consensus.cc:1226] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.621803 13816 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:15.622099 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.623067 16053 raft_consensus.cc:686] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:15.615522 15976 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:15.613899 16231 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:15.624338 16231 raft_consensus.cc:686] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:15.623950 16053 consensus_queue.cc:237] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.625646 16285 raft_consensus.cc:3032] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:15.625382 16231 consensus_queue.cc:237] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.625721 16254 consensus_queue.cc:1046] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:15.626822 16254 raft_consensus.cc:3032] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:15.627131 16254 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:15.592510 13805 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:15.601984 16147 consensus_queue.cc:237] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:15.617373 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 206397us
I20260807 08:01:15.623684 13661 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.628003 16254 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.629181 16254 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.630019 16267 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:15.630999 15976 raft_consensus.cc:686] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:15.631259 16267 consensus_queue.cc:1046] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:15.632431 16230 raft_consensus.cc:3032] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:15.632786 16230 raft_consensus.cc:729] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:15.632858 16278 raft_consensus.cc:2726] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:15.633642 16230 consensus_queue.cc:260] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.626144 16285 raft_consensus.cc:729] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:15.635288 16285 consensus_queue.cc:260] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.631971 15976 consensus_queue.cc:237] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.602500 13702 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:15.635697 13769 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:15.601413 16243 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:15.613047 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 107268us
I20260807 08:01:15.636785 13661 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.639467 14583 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:15.641362 14580 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.619141 16015 raft_consensus.cc:2726] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:15.644317 14583 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:15.643590 13769 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:15.642752 13537 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.649260 13661 leader_election.cc:398] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:15.649837 13661 leader_election.cc:401] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:15.642359 15891 consensus_queue.cc:1046] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260807 08:01:15.651667 16278 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:15.648418 16105 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:15.654428 13537 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:15.640121 16285 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:15.661453 13585 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:15.662130 16231 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:15.662634 16231 raft_consensus.cc:686] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:15.662858 16230 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:15.663074 13588 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:15.663478 16231 consensus_queue.cc:237] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.664116 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.666025 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.666338 15891 raft_consensus.cc:3032] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:15.666741 13816 raft_consensus.cc:2379] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:15.667490 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:15.668062 13587 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:15.648554 16130 consensus_queue.cc:1046] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260807 08:01:15.670243 13540 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:15.644415 16255 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:15.671411 13769 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:15.648610 16243 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:15.676350 16243 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:15.676823 16243 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:15.659261 16097 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:15.677713 16097 raft_consensus.cc:686] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:15.678639 16097 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.680624 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:15.653462 16279 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:15.681610 16215 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:15.682354 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:15.681751 16279 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:15.683228 13804 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:15.683336 16279 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:15.686405 13805 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260807 08:01:15.686918 13805 pending_rounds.cc:76] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 5
I20260807 08:01:15.654285 16015 raft_consensus.cc:2741] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260807 08:01:15.688323 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 44077.9us
I20260807 08:01:15.691272 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:15.698354 16105 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:15.685565 16224 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:15.692099 16293 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:15.699302 16253 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:15.687251 13805 pending_rounds.cc:97] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260807 08:01:15.842229 13805 raft_consensus.cc:2866] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:15.666761 15891 raft_consensus.cc:729] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:15.824824 13769 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
W20260807 08:01:15.837841 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 148304us
I20260807 08:01:15.844199 14583 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.847463 13769 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:15.822335 16267 raft_consensus.cc:3032] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
W20260807 08:01:15.855471 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 164230us
I20260807 08:01:15.845392 16253 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:15.856482 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
W20260807 08:01:15.902760 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 81026.8us
I20260807 08:01:15.843751 16195 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260807 08:01:15.851490 16176 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:15.856372 16015 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:15.851549 16267 raft_consensus.cc:729] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:15.919041 16015 raft_consensus.cc:686] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:15.920151 16015 consensus_queue.cc:237] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.919894 16267 consensus_queue.cc:260] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:15.856447 16222 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:15.922564 16222 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:15.922996 16222 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.925889 13805 raft_consensus.cc:1226] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:15.927887 16253 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.918231 16279 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:15.930831 13661 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.932632 13661 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.903486 16224 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:15.939769 16224 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:15.940171 16224 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:15.910701 16243 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:15.941421 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 85017.6us
I20260807 08:01:15.951731 13661 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.954547 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:15.977383 13661 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.978459 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:15.936571 16230 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:15.987304 13769 leader_election.cc:398] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:15.988258 13540 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:15.943655 16139 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:15.952343 16147 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:15.936285 16105 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260807 08:01:15.942977 16267 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:15.998487 13661 leader_election.cc:398] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:16.005798 13661 leader_election.cc:401] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260807 08:01:15.986280 16053 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:15.989207 16231 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:15.974947 16085 consensus_queue.cc:1046] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260807 08:01:16.016981 13769 leader_election.cc:398] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:16.016687 13661 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.018853 13769 leader_election.cc:401] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260807 08:01:16.017720 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 163453us
I20260807 08:01:15.996666 16147 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:16.014140 16297 log.cc:924] Time spent T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260807 08:01:16.009899 16167 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:16.010056 16269 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:15.950443 15891 consensus_queue.cc:260] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.017306 16231 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:16.036340 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 184228us
I20260807 08:01:16.023819 16147 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.037992 13814 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:16.020184 16280 raft_consensus.cc:2726] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:16.025211 16234 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.032090 16269 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:16.081583 13587 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:16.083022 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:16.031814 16285 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
W20260807 08:01:16.105701 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 50367.1us
W20260807 08:01:16.086961 16294 log.cc:924] Time spent T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260807 08:01:16.049701 16231 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:16.057729 16298 log.cc:924] Time spent T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260807 08:01:16.065498 13769 leader_election.cc:398] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:16.115782 13769 leader_election.cc:401] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:16.058495 16279 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:16.111835 13661 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.083379 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:16.149731 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 94990.7us
I20260807 08:01:16.152350 16126 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:16.158092 16172 raft_consensus.cc:3032] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:16.158525 16172 raft_consensus.cc:729] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:16.159394 16172 consensus_queue.cc:260] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.160686 16172 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.116801 16243 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:16.165642 16230 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 08:01:16.166712 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 47329.1us
I20260807 08:01:16.168181 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:16.169658 16280 consensus_queue.cc:1046] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:16.170871 16280 consensus_queue.cc:1046] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:16.107661 16269 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.171942 16280 raft_consensus.cc:3032] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:16.172343 16280 raft_consensus.cc:729] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:16.080711 16139 consensus_queue.cc:1046] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260807 08:01:16.058204 16167 consensus_queue.cc:1035] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260807 08:01:16.171128 16279 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 08:01:16.174165 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 175295us
I20260807 08:01:16.190514 13540 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:16.192395 13540 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.193627 16085 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:16.194115 16085 raft_consensus.cc:686] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:16.194819 13540 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.086953 15976 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260807 08:01:16.113484 16310 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260807 08:01:16.169435 13661 leader_election.cc:398] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:16.184651 16243 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:16.110664 15891 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:16.194972 16085 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.175480 13816 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:16.219584 16147 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.209975 13661 leader_election.cc:398] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:16.224006 16224 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.225328 13589 raft_consensus.cc:1226] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:16.226293 16222 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.227650 16289 raft_consensus.cc:3032] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:16.228008 16289 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:16.228363 16222 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:16.229156 13807 raft_consensus.cc:1226] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:16.228780 16289 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:16.215185 16144 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:16.211071 16243 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.230239 16105 consensus_queue.cc:1046] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:16.230376 16276 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:16.232020 16306 consensus_queue.cc:1046] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:16.232967 16234 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:16.233234 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:16.161123 16253 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:16.236685 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 155431us
I20260807 08:01:16.240938 16289 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:16.253829 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 102045us
I20260807 08:01:16.173244 16280 consensus_queue.cc:260] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:16.257369 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 62616.8us
I20260807 08:01:16.260891 13540 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.240130 16285 raft_consensus.cc:686] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:16.270658 13807 raft_consensus.cc:1261] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:16.271271 16285 consensus_queue.cc:237] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:16.253813 16147 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:16.277829 13805 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:16.279706 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:16.283208 13769 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:16.292894 16224 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:16.294034 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:16.294533 16231 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.294710 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.294853 16321 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 08:01:16.296954 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 63862.5us
I20260807 08:01:16.298545 16315 raft_consensus.cc:2781] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:16.299227 16315 raft_consensus.cc:686] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:16.299330 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:16.300246 16315 consensus_queue.cc:237] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:16.290042 16267 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:16.301837 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.302044 16193 raft_consensus.cc:3032] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:16.302747 14623 raft_consensus.cc:2379] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 2: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:16.277725 16313 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:16.305357 16313 raft_consensus.cc:686] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:16.302549 16193 raft_consensus.cc:729] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:16.306391 16313 consensus_queue.cc:237] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:16.302510 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:16.293251 16253 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:16.298503 13661 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.310545 13589 raft_consensus.cc:1226] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:16.297210 14580 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.286070 16243 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:16.284806 16216 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:16.324568 16310 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:16.324817 13700 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:16.325189 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:16.325763 16130 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:16.332015 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.296751 14622 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.340727 13428 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56596 (request call id 306) took 1107 ms. Trace:
I20260807 08:01:16.341200 13428 rpcz_store.cc:276] 0807 08:01:15.233644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:15.233837 (+ 193us) service_pool.cc:227] Handling call
0807 08:01:16.340635 (+1106798us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:15.671121 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:15.719217 (+ 48096us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:15.719269 (+ 52us) write_op.cc:435] Acquired schema lock
0807 08:01:15.719293 (+ 24us) tablet.cc:660] Decoding operations
0807 08:01:15.846521 (+127228us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:15.853885 (+ 7364us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:15.854953 (+ 1068us) tablet.cc:683] Acquiring locks for 6 operations
0807 08:01:15.928092 (+ 73139us) tablet.cc:699] Row locks acquired
0807 08:01:15.928117 (+ 25us) write_op.cc:260] PREPARE: finished
0807 08:01:15.928261 (+ 144us) write_op.cc:270] Start()
0807 08:01:15.928382 (+ 121us) write_op.cc:276] Timestamp: P: 1786089675928223 usec, L: 0
0807 08:01:15.928408 (+ 26us) op_driver.cc:331] REPLICATION: starting
0807 08:01:15.929109 (+ 701us) log.cc:842] Serialized 2598 byte log entry
0807 08:01:16.057530 (+128421us) op_driver.cc:444] REPLICATION: finished
0807 08:01:16.319635 (+262105us) write_op.cc:301] APPLY: starting
0807 08:01:16.319720 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:16.319922 (+ 202us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:16.321273 (+ 1351us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:16.325183 (+ 3910us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:16.325397 (+ 214us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0807 08:01:16.325444 (+ 47us) write_op.cc:312] APPLY: finished
0807 08:01:16.325739 (+ 295us) log.cc:842] Serialized 53 byte log entry
0807 08:01:16.325911 (+ 172us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:16.326032 (+ 121us) write_op.cc:454] Released schema lock
0807 08:01:16.326079 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:16.326385 (+ 306us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:16.326414 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":214994,"apply.run_cpu_time_us":3436,"apply.run_wall_time_us":7163,"num_ops":6,"prepare.queue_time_us":210438,"prepare.run_cpu_time_us":8344,"prepare.run_wall_time_us":281589,"replication_time_us":108373,"thread_start_us":284210,"threads_started":2}]]}
I20260807 08:01:16.352381 13661 leader_election.cc:398] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:16.353292 13661 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.354094 13705 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:16.357774 13769 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.360455 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.360311 16269 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:16.361150 16126 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:16.363087 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:16.363263 13769 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.363513 13661 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:16.325448 16176 raft_consensus.cc:3032] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:16.306916 16193 consensus_queue.cc:260] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.365932 16253 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:16.371500 16015 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:16.369791 13432 catalog_manager.cc:5495] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 3, leader changed from 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:16.371898 13661 leader_election.cc:398] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:16.372323 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.372429 16316 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:16.327067 15976 consensus_queue.cc:1046] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260807 08:01:16.379174 13432 catalog_manager.cc:5495] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 3, leader changed from 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:16.358791 16130 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:16.374990 16167 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:16.366181 16176 raft_consensus.cc:729] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:16.382611 16126 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:16.389964 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.333441 16216 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.398428 13705 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:16.404886 13661 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.406667 13769 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.400316 16193 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.397686 16269 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:16.419169 13661 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:16.422915 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:16.426712 16176 consensus_queue.cc:260] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.428735 16176 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.430342 16273 raft_consensus.cc:3032] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:16.430433 13702 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:16.431061 16273 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:16.409658 16130 consensus_queue.cc:1046] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260807 08:01:16.434845 16267 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:16.435415 16267 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:16.435882 16267 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.432386 16273 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.393419 16253 consensus_queue.cc:1046] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
W20260807 08:01:16.461400 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 205141us
I20260807 08:01:16.412483 16122 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260807 08:01:16.460763 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 199965us
I20260807 08:01:16.413035 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:16.414181 16139 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 08:01:16.509680 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 46436us
I20260807 08:01:16.445941 14623 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 2: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
W20260807 08:01:16.511703 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 67444.5us
I20260807 08:01:16.456869 16285 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:16.458104 16130 raft_consensus.cc:3032] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:16.410764 16167 consensus_queue.cc:1046] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260807 08:01:16.520895 13661 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.456899 16231 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:16.524260 13540 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.539242 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
W20260807 08:01:16.676011 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 136889us
I20260807 08:01:16.522116 16280 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.511327 16015 consensus_queue.cc:1046] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260807 08:01:16.530429 16172 consensus_queue.cc:1046] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260807 08:01:16.525689 16130 raft_consensus.cc:729] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:16.553535 16320 log.cc:924] Time spent T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
W20260807 08:01:16.622242 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 101460us
I20260807 08:01:16.530042 16085 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260807 08:01:16.514076 16126 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260807 08:01:16.523242 16285 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:16.525130 16273 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:16.443221 13805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
W20260807 08:01:16.621966 16326 log.cc:924] Time spent T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.104s user 0.000s sys 0.003s
I20260807 08:01:16.619300 14622 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:16.621006 13701 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:16.521924 13700 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:16.668919 16316 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:16.672411 16267 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.525014 16122 raft_consensus.cc:3032] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:16.554204 16253 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
W20260807 08:01:16.680745 16343 log.cc:924] Time spent T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.057s user 0.003s sys 0.000s
W20260807 08:01:16.683707 16344 log.cc:924] Time spent T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.062s user 0.000s sys 0.002s
I20260807 08:01:16.788455 16267 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:16.788862 13585 raft_consensus.cc:1226] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
W20260807 08:01:16.688501 13769 leader_election.cc:334] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.507s (SENT)
I20260807 08:01:16.688681 16144 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:16.689862 16216 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:16.789710 13769 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:16.781991 16289 consensus_meta.cc:287] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.523s user 0.006s sys 0.008s
W20260807 08:01:16.692952 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 168780us
I20260807 08:01:16.790416 13580 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:16.790812 13540 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.785372 16122 raft_consensus.cc:729] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:01:16.699867 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 156437us
I20260807 08:01:16.793110 16122 consensus_queue.cc:260] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.794225 16122 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:01:16.794678 14583 leader_election.cc:334] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.680s (SENT)
I20260807 08:01:16.709333 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:16.713079 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.714103 13590 raft_consensus.cc:1226] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:16.773993 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:16.774048 13805 raft_consensus.cc:595] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:16.780197 13588 raft_consensus.cc:1226] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260807 08:01:16.793111 13540 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.780280 15891 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:16.775292 16139 raft_consensus.cc:3032] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:16.775849 14630 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:16.781744 13589 raft_consensus.cc:1226] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:16.774259 13584 raft_consensus.cc:1226] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:16.725965 16346 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
W20260807 08:01:16.719656 16341 log.cc:924] Time spent T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.101s user 0.002s sys 0.000s
I20260807 08:01:16.795209 14583 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.725222 16285 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:16.803442 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.800997 16172 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:16.804452 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.805665 16325 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:16.809656 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
W20260807 08:01:16.811136 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 133944us
I20260807 08:01:16.816712 14629 raft_consensus.cc:1226] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:16.814451 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:16.817106 14630 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 2: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:16.818640 13590 raft_consensus.cc:1226] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:16.832582 14629 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:16.801182 16144 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:16.837930 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.838095 16130 consensus_queue.cc:260] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.801666 14623 raft_consensus.cc:1226] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:16.839941 14630 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:16.840369 16325 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:16.841598 16254 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:16.806712 15891 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.842643 16325 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:16.844241 15891 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:16.844353 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.844969 16346 raft_consensus.cc:686] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:16.801448 16139 raft_consensus.cc:729] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:16.846074 16346 consensus_queue.cc:237] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.846356 16139 consensus_queue.cc:260] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.847604 16139 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:16.849593 14622 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:16.849818 16224 raft_consensus.cc:3032] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:16.831625 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:16.850373 16224 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:16.850937 13579 raft_consensus.cc:2379] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:16.851272 13592 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:16.851383 16224 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:16.839540 16130 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:16.864805 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:16.865557 14583 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.866647 13587 raft_consensus.cc:2354] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:16.871850 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:16.874796 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.874981 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:16.831597 16105 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:16.882616 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:16.885097 14583 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.878916 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:16.823339 16280 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:16.881151 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:16.881908 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:16.905666 16193 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:16.908510 13709 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:16.909970 13701 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:16.910996 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:16.911419 13701 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:16.913811 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:16.914467 16267 consensus_queue.cc:1046] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:16.915647 13579 raft_consensus.cc:1226] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:16.915544 16267 consensus_queue.cc:1046] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:16.916574 16276 raft_consensus.cc:3032] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:16.916921 16276 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:16.917148 13587 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:16.909621 16280 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:16.949702 16234 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:16.929340 13585 raft_consensus.cc:1226] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:16.919178 16276 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:16.917806 16243 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:16.923038 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:16.990513 13708 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:16.928202 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:16.926111 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:16.992025 13710 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:16.917703 16289 raft_consensus.cc:3032] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:16.919862 13700 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:16.929502 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:16.867748 16306 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:16.870505 16172 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:16.870195 15891 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260807 08:01:16.886530 13588 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:16.890733 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:16.961081 14623 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:16.962452 14625 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:16.964866 16222 raft_consensus.cc:3032] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:16.965013 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:16.918608 13661 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.922369 13709 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:16.943065 16139 consensus_queue.cc:1046] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:16.944733 14622 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:16.969015 13592 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:16.971412 13590 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:16.965225 14626 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:16.891700 14583 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.000504 13588 raft_consensus.cc:595] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:16.987195 13587 raft_consensus.cc:1226] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260807 08:01:16.993099 13702 raft_consensus.cc:2362] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4. Re-sending same reply.
I20260807 08:01:16.870414 16316 consensus_queue.cc:1046] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20260807 08:01:16.995410 13584 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 2: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:16.994058 16289 raft_consensus.cc:729] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:17.005692 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 87162.8us
I20260807 08:01:16.988809 16285 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.007105 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.000767 16276 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:17.005420 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:16.890187 13540 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:17.009892 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 118299us
I20260807 08:01:16.998929 16172 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.005065 16222 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:17.017946 14583 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:16.999478 16306 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.008958 16267 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:17.011232 15976 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:17.020118 13814 raft_consensus.cc:1261] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 08:01:17.020720 16216 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:17.029007 13580 raft_consensus.cc:2379] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 2: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:17.031798 16216 raft_consensus.cc:686] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:17.020947 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.026674 16280 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.021701 16313 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:17.033213 16313 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:17.032965 16216 consensus_queue.cc:237] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.033763 16313 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:17.021821 13817 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:17.033895 16222 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.022168 13812 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:17.022559 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:17.035451 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.023140 16360 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:17.036605 16285 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:17.036509 16360 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.039157 16360 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.023329 16289 consensus_queue.cc:260] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.024238 16130 raft_consensus.cc:3032] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:17.039868 16130 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:17.040423 16289 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:17.040681 16130 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.025799 13706 raft_consensus.cc:2379] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:17.008462 16243 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:17.057461 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:16.990515 16234 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.009624 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:17.018867 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 128799us
I20260807 08:01:17.059679 13540 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.030895 16139 consensus_queue.cc:1046] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:17.042397 16306 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:17.043701 14583 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.054463 16346 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.014351 16359 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:17.050321 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 227623us
I20260807 08:01:17.050897 15976 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:17.050000 13710 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:17.053988 13708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:01:17.040593 16172 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.060801 13540 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.097815 13811 raft_consensus.cc:1261] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260807 08:01:17.084656 13432 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56582 (request call id 304) took 1702 ms. Trace:
I20260807 08:01:17.098271 13811 pending_rounds.cc:76] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 1
I20260807 08:01:17.090848 13430 catalog_manager.cc:5495] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:17.096155 13708 raft_consensus.cc:595] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:17.098754 13811 pending_rounds.cc:97] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260807 08:01:17.098886 13708 raft_consensus.cc:603] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:17.099154 13811 raft_consensus.cc:2866] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:17.097491 13710 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:17.098505 13432 rpcz_store.cc:276] 0807 08:01:15.382468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:15.382625 (+ 157us) service_pool.cc:227] Handling call
0807 08:01:17.084572 (+1701947us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:16.420710 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:16.441610 (+ 20900us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:16.441647 (+ 37us) write_op.cc:435] Acquired schema lock
0807 08:01:16.441659 (+ 12us) tablet.cc:660] Decoding operations
0807 08:01:16.442058 (+ 399us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:16.442110 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:16.442137 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0807 08:01:16.442519 (+ 382us) tablet.cc:699] Row locks acquired
0807 08:01:16.442534 (+ 15us) write_op.cc:260] PREPARE: finished
0807 08:01:16.442672 (+ 138us) write_op.cc:270] Start()
0807 08:01:16.442786 (+ 114us) write_op.cc:276] Timestamp: P: 1786089676442626 usec, L: 0
0807 08:01:16.442807 (+ 21us) op_driver.cc:331] REPLICATION: starting
0807 08:01:16.554331 (+111524us) log.cc:842] Serialized 919 byte log entry
0807 08:01:17.038477 (+484146us) op_driver.cc:444] REPLICATION: finished
0807 08:01:17.070597 (+ 32120us) write_op.cc:301] APPLY: starting
0807 08:01:17.070707 (+ 110us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:17.070871 (+ 164us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:17.071020 (+ 149us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:17.071489 (+ 469us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:17.071703 (+ 214us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0807 08:01:17.071744 (+ 41us) write_op.cc:312] APPLY: finished
0807 08:01:17.072009 (+ 265us) log.cc:842] Serialized 29 byte log entry
0807 08:01:17.072212 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:17.072321 (+ 109us) write_op.cc:454] Released schema lock
0807 08:01:17.072357 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:17.072641 (+ 284us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:17.092226 (+ 19585us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":31821,"apply.run_cpu_time_us":2524,"apply.run_wall_time_us":22040,"num_ops":2,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":9136,"prepare.run_wall_time_us":286872,"replication_time_us":595561,"thread_start_us":31483,"threads_started":1}]]}
W20260807 08:01:17.093971 13769 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.840s (SENT) [suppressed 66 similar messages]
I20260807 08:01:17.060544 16224 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:17.103091 13769 leader_election.cc:334] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.869s (SENT)
I20260807 08:01:17.103223 13430 catalog_manager.cc:5495] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:17.103816 13769 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.064368 16269 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.065292 16176 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.110538 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.112776 13710 raft_consensus.cc:2362] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4. Re-sending same reply.
I20260807 08:01:17.114460 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.115078 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:17.105448 13430 catalog_manager.cc:5495] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 2 to 3, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 3 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260807 08:01:17.111231 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 50490.9us
I20260807 08:01:17.102890 16085 raft_consensus.cc:2726] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.123342 13710 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:17.062279 16267 consensus_queue.cc:1046] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
W20260807 08:01:17.123499 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 67874.1us
I20260807 08:01:17.060331 16243 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.107218 16361 raft_consensus.cc:3032] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:17.121943 16358 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:17.126421 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 08:01:17.107275 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.133464 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:17.134311 14630 raft_consensus.cc:2379] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 2: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:17.136366 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.137154 13584 raft_consensus.cc:2379] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:17.097997 16359 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.138636 16222 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:17.139261 16234 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:17.139668 16359 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:17.141769 13418 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 0248ade936924e628c46b8de6252af1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 9210dbc4cc9c45e5b26a463b705d362c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:17.107237 16368 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.144313 16358 raft_consensus.cc:686] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:17.145710 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.146450 13807 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:17.130125 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.149879 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.127992 13540 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.111912 16193 consensus_queue.cc:1046] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260807 08:01:17.123574 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.131667 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.101891 15976 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.132264 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:17.153164 13811 raft_consensus.cc:2379] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:17.149853 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.130597 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:17.154021 13710 raft_consensus.cc:2354] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4.
I20260807 08:01:17.095304 16346 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.156972 13584 raft_consensus.cc:1261] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 08:01:17.157605 16139 consensus_queue.cc:1035] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:17.157869 16346 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:17.158972 13657 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:17.160528 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.123148 13430 catalog_manager.cc:5495] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 3, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 3 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:17.161372 13540 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.170781 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.171850 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.173879 13661 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:17.171464 13710 raft_consensus.cc:2362] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4. Re-sending same reply.
I20260807 08:01:17.176167 13705 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:17.177984 13661 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.178227 16144 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:17.180518 16144 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:17.129374 16354 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:17.128759 16361 raft_consensus.cc:729] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:17.186019 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.186376 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.186975 13705 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:17.187688 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:17.189416 13769 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:17.184571 13661 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.185747 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.169647 16172 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:17.169777 16267 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:17.199003 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:17.202277 13661 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.184016 16360 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.183889 16234 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:17.193882 13588 raft_consensus.cc:2379] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:17.209017 13814 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:17.226560 13703 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
W20260807 08:01:17.213681 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 52368.9us
I20260807 08:01:17.214644 16172 raft_consensus.cc:2726] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:17.201920 15891 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:17.207340 16015 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:17.198416 16253 consensus_queue.cc:1046] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260807 08:01:17.220208 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.145202 16358 consensus_queue.cc:237] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.196982 13812 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:17.231958 13540 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.234402 13710 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:17.242901 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.194978 16176 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.175176 16195 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260807 08:01:17.239010 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.183676 16122 consensus_queue.cc:1046] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260807 08:01:17.205804 16267 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:17.201658 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.244385 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.221807 16285 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:17.251374 13710 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:17.223567 16289 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:17.198539 16354 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.214082 16105 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260807 08:01:17.222414 16234 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.311993 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.195060 16368 consensus_queue.cc:1046] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260807 08:01:17.231395 16346 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
W20260807 08:01:17.314670 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 70360.1us
W20260807 08:01:17.315478 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 56501.3us
I20260807 08:01:17.320919 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.245445 16377 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.317094 16385 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.239610 15891 raft_consensus.cc:686] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:17.246378 16172 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 08:01:17.322769 13769 leader_election.cc:334] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.918s (SENT)
I20260807 08:01:17.320556 13540 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.356619 13769 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.316459 16267 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:17.324599 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.358784 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.359396 13710 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:17.325112 16130 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:17.234427 16360 consensus_queue.cc:1035] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20260807 08:01:17.245646 16147 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:17.321588 13703 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
W20260807 08:01:17.347872 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 186344us
W20260807 08:01:17.349997 16376 log.cc:924] Time spent T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
W20260807 08:01:17.354517 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 206961us
I20260807 08:01:17.322561 15976 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.326555 13701 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:17.364341 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.255322 16176 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:17.371999 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 49238.8us
W20260807 08:01:17.372376 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 217761us
I20260807 08:01:17.360349 16269 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
W20260807 08:01:17.378804 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 174388us
I20260807 08:01:17.393095 13540 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.364936 16346 raft_consensus.cc:686] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:17.373027 13769 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:17.233968 16361 consensus_queue.cc:260] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.396703 13703 raft_consensus.cc:1261] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:17.401597 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:17.368132 16243 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.397364 13710 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:17.356603 16313 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.409385 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:17.417273 13589 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:17.425927 13703 raft_consensus.cc:1226] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:17.404510 16316 raft_consensus.cc:3032] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:17.403862 16085 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:17.374217 16172 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260807 08:01:17.407461 16315 raft_consensus.cc:3032] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:17.355505 16377 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:17.435526 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 62568.6us
I20260807 08:01:17.437592 16315 raft_consensus.cc:729] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:17.442804 13769 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:17.438956 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:17.451762 13769 leader_election.cc:398] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:17.429370 16316 raft_consensus.cc:729] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:17.439792 13701 raft_consensus.cc:1226] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:17.397423 15891 consensus_queue.cc:237] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.463672 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.464581 16377 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.465139 16377 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:17.466297 13706 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:17.466586 16384 log.cc:924] Time spent T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260807 08:01:17.467602 13703 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 08:01:17.468396 16387 log.cc:924] Time spent T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260807 08:01:17.470234 16313 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:17.467628 13706 raft_consensus.cc:1261] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 08:01:17.474242 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:17.474747 16379 log.cc:924] Time spent T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260807 08:01:17.476892 13700 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:17.474265 16404 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:17.480680 16404 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:17.481050 16404 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:17.474275 16234 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:17.486830 16276 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.487322 16276 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.488346 16361 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:17.489276 16276 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.489887 16289 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.490437 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.495002 16306 raft_consensus.cc:3032] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:17.495544 16306 raft_consensus.cc:729] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:17.498158 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.498296 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.502024 16255 raft_consensus.cc:3032] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:17.502521 16255 raft_consensus.cc:729] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:17.502090 16346 consensus_queue.cc:237] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.503311 16255 consensus_queue.cc:260] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.504478 16255 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:17.486277 13702 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:17.565659 13540 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.496589 16306 consensus_queue.cc:260] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:17.566777 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 93160.4us
I20260807 08:01:17.456230 16085 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:17.446483 16253 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:17.406090 16385 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260807 08:01:17.573959 16172 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.405625 16401 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260807 08:01:17.472872 16325 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election won for term 5
W20260807 08:01:17.536324 16343 log.cc:924] Time spent T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260807 08:01:17.578434 13709 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:17.580338 13817 raft_consensus.cc:1261] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 08:01:17.582167 13540 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.575738 16234 consensus_queue.cc:1046] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.015s
I20260807 08:01:17.591310 16172 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.596846 16172 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.599565 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.600564 13817 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:17.586108 16407 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:17.602936 16407 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:17.603292 16407 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:17.605378 16316 consensus_queue.cc:260] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.611311 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.576722 16015 raft_consensus.cc:686] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:17.575078 16269 raft_consensus.cc:686] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:17.618610 13709 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:17.619258 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.448462 16315 consensus_queue.cc:260] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.620498 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.621989 13816 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:17.578624 16254 consensus_queue.cc:1046] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260807 08:01:17.624409 13657 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:17.625473 13540 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:17.608830 13711 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:17.611630 16354 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:17.631067 16414 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:17.626330 16176 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.631569 16414 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:17.623155 16306 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:17.626302 16234 raft_consensus.cc:3032] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:17.626866 16401 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.631973 16414 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:17.629979 16147 raft_consensus.cc:2726] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:17.639461 16147 consensus_queue.cc:1046] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260807 08:01:17.639688 16419 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:17.636122 16172 consensus_queue.cc:1035] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260807 08:01:17.642002 16419 raft_consensus.cc:686] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:17.642817 16172 raft_consensus.cc:3032] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:17.643227 16172 raft_consensus.cc:729] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:17.643090 16419 consensus_queue.cc:237] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.647293 16254 consensus_queue.cc:1046] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:17.647225 16015 consensus_queue.cc:237] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.648413 16254 raft_consensus.cc:3032] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:17.649448 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.650094 13710 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:17.627669 16417 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:17.648788 16254 raft_consensus.cc:729] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:17.644404 16172 consensus_queue.cc:260] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.637603 16234 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:17.676688 13709 raft_consensus.cc:1261] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260807 08:01:17.689663 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.689796 13769 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:17.692330 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 62575.5us
I20260807 08:01:17.656066 16267 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.690023 16253 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:17.693744 16377 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.699501 13540 leader_election.cc:398] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4.
I20260807 08:01:17.705777 13705 raft_consensus.cc:2354] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3.
I20260807 08:01:17.706702 16085 raft_consensus.cc:686] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:17.690416 13588 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:17.657714 16269 consensus_queue.cc:237] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.670797 16417 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.710115 13540 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:17.708212 16346 consensus_queue.cc:1035] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:17.694929 16234 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.685065 16414 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.708211 16255 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:17.705318 16377 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.728847 13537 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:17.692653 16254 consensus_queue.cc:260] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.730711 16285 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:17.730036 16234 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:17.719964 16325 raft_consensus.cc:686] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:17.734637 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.724884 16255 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:17.738405 13710 raft_consensus.cc:2354] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:17.738902 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.740842 16253 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:17.741402 16423 log.cc:924] Time spent T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260807 08:01:17.742014 16385 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:17.742550 16385 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.742954 16412 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.743316 16267 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.736617 16285 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:17.744122 13540 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.744625 16385 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:17.744875 16412 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:17.745353 16412 raft_consensus.cc:686] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:17.746369 16412 consensus_queue.cc:237] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.747009 16377 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.748517 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.748790 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:17.749300 13809 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:17.750101 16436 log.cc:924] Time spent T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260807 08:01:17.751539 16431 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:17.753962 16411 raft_consensus.cc:3032] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:17.741349 16253 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.757728 13811 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:17.694553 16425 consensus_queue.cc:1046] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260807 08:01:17.762869 13769 leader_election.cc:398] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:17.764127 13769 leader_election.cc:401] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:17.760536 13809 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:17.760839 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.767774 13657 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:17.731698 16417 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:17.769279 13537 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.721295 16404 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.733188 16440 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:17.754437 16411 raft_consensus.cc:729] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:17.758566 16255 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:17.773820 16422 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 6
W20260807 08:01:17.798755 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 391811us
I20260807 08:01:17.777163 16429 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.777979 16254 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:17.778781 16377 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:17.788520 16253 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:17.790699 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:17.819986 16427 log.cc:924] Time spent T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260807 08:01:17.791564 16440 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:17.772953 16401 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260807 08:01:17.830096 16361 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.769761 16385 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260807 08:01:17.835736 14583 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.837633 14583 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.830641 16361 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.840062 16404 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:17.840426 16393 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.840951 16403 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.841184 16361 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:17.832777 16435 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.841548 16403 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.842191 16435 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.846589 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:17.846891 16195 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:17.848045 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.849246 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:17.854552 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:17.855209 13657 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:17.820952 16412 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260807 08:01:17.854743 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:17.861228 13710 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:17.832746 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:17.822008 16377 raft_consensus.cc:686] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:17.805125 16243 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260807 08:01:17.866585 14626 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:17.867060 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.868078 13809 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:17.831815 16440 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:17.859522 14583 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.874397 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:17.857362 16361 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:17.771021 16085 consensus_queue.cc:237] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.869620 13657 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:17.870674 14629 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:17.876255 14630 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:17.877197 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:17.877933 14580 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.878825 16195 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:17.832818 16437 raft_consensus.cc:2741] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 08:01:17.890727 16195 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 08:01:17.891233 16442 log.cc:924] Time spent T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20260807 08:01:17.891376 16416 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:17.892041 16377 consensus_queue.cc:237] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.863548 16285 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.894871 16193 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:17.875912 16176 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:17.875742 16361 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:17.896154 16243 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:17.882141 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:17.899251 13703 raft_consensus.cc:1261] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:17.902442 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:17.893266 16435 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.897876 16385 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.901609 16403 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.812215 16429 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.822693 16325 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.865373 16421 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260807 08:01:17.922246 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:17.912999 16193 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:17.821420 16411 consensus_queue.cc:260] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.941763 13657 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:17.902283 16414 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:17.943572 16414 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:17.904423 16416 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:17.944654 16416 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:17.923751 13588 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:17.939312 13710 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:17.920645 16361 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:17.947090 13769 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:17.947634 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 70510.6us
I20260807 08:01:17.900588 16407 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.949009 14583 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:17.957209 16172 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:17.963450 16085 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:17.964397 16085 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.965291 16431 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:17.965906 16431 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:17.966779 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.967684 13584 raft_consensus.cc:2379] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:17.968168 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.968693 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.969103 13587 raft_consensus.cc:2379] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:17.840997 16393 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:17.940858 16193 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:01:17.955273 16378 log.cc:924] Time spent T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.595s user 0.006s sys 0.000s
I20260807 08:01:17.983731 16085 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:17.998435 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:17.957825 16400 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:17.927815 16385 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:17.971060 16404 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:17.979105 16216 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:17.976452 16403 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:17.930222 16222 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260807 08:01:18.009073 14619 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:18.003227 16400 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
W20260807 08:01:18.021874 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 246597us
I20260807 08:01:18.023020 13540 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:18.013933 16452 log.cc:924] Time spent T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.149s user 0.000s sys 0.002s
I20260807 08:01:18.014070 16411 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:18.027128 13540 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.014298 16393 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.014437 16404 raft_consensus.cc:686] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:18.032258 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:18.032829 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 155021us
I20260807 08:01:18.033380 13590 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:18.033557 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.035133 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.015635 16216 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.036098 13587 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:18.025434 16400 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:18.037580 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.023401 16403 raft_consensus.cc:686] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:18.043249 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.050524 13591 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:18.029980 16313 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:18.036994 14583 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:18.040472 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.059772 13584 raft_consensus.cc:2379] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:18.043548 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.047494 16285 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.044160 14580 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:18.061862 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:18.026975 16085 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260807 08:01:18.032240 16455 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:18.064198 14630 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:18.067451 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.079478 13814 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:18.068341 16306 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:18.068732 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.080592 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.081130 13809 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:18.068995 16385 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.030009 16224 consensus_queue.cc:1046] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260807 08:01:18.069768 16459 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:18.048270 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:18.073911 13805 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:18.074221 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.086546 16243 raft_consensus.cc:2726] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:18.087625 13661 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:18.089093 16414 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:18.089768 16461 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.090171 16407 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.090592 16461 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.073998 16404 consensus_queue.cc:237] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.097003 14580 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:18.075652 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.076308 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.103477 13808 raft_consensus.cc:2396] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 08:01:18.081959 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:18.095772 13805 raft_consensus.cc:1226] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260807 08:01:18.106856 16454 log.cc:924] Time spent T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
W20260807 08:01:18.108779 16458 log.cc:924] Time spent T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20260807 08:01:18.109206 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:18.096061 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 326905us
I20260807 08:01:18.098687 14580 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:18.030683 16255 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.100015 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:18.102101 16462 log.cc:924] Time spent T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20260807 08:01:18.066195 16435 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.102315 16393 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:18.149921 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.150259 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:18.098356 13817 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:18.114858 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.152606 14580 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:18.154346 16176 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:18.154945 16176 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:18.155553 16176 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:18.127550 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:18.067212 16421 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:18.133756 14630 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:18.162014 13661 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:18.165300 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:18.171288 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:18.159772 16306 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260807 08:01:18.172729 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:18.173646 16216 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:18.149048 16243 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:18.175406 16243 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:18.175877 16243 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:18.175964 16431 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.165330 15976 consensus_queue.cc:1046] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260807 08:01:18.187244 16216 raft_consensus.cc:3032] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:18.185702 16431 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.187897 16216 raft_consensus.cc:729] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:18.189838 16216 consensus_queue.cc:260] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.191610 16461 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.191906 16216 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:18.197305 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.176016 16255 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:18.110015 16429 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.161028 16421 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:18.149873 16393 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:18.159941 16455 raft_consensus.cc:686] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:18.225047 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.230020 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.238713 13580 raft_consensus.cc:2379] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 2: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:18.227164 16393 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:18.229075 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.246454 13808 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:18.247483 16455 consensus_queue.cc:237] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.249424 16176 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.125504 16222 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20260807 08:01:18.149175 16122 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260807 08:01:18.258832 16361 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.263258 14580 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.159979 16425 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260807 08:01:18.270267 16105 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:18.272846 16105 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:18.273557 16105 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:18.273680 16176 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.275211 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.276480 14623 raft_consensus.cc:2379] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:18.276978 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:18.278074 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.162503 16435 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.230648 16459 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:18.278357 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:18.226225 16421 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:18.326396 13580 raft_consensus.cc:2362] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3. Re-sending same reply.
I20260807 08:01:18.280053 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:18.323412 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:18.324283 14630 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:18.279196 13585 raft_consensus.cc:1226] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:18.278100 13587 raft_consensus.cc:1226] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:18.282281 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:18.333408 13540 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.335536 16377 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:18.336391 16377 raft_consensus.cc:686] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:18.340453 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 178529us
I20260807 08:01:18.343573 13580 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:18.278827 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:18.343513 16401 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:18.194283 16403 consensus_queue.cc:237] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.350114 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:18.358476 13592 raft_consensus.cc:1261] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:18.353428 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:18.353574 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:18.356402 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1589) took 1470 ms. Trace:
I20260807 08:01:18.360353 13589 rpcz_store.cc:276] 0807 08:01:16.885820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:16.888446 (+ 2626us) service_pool.cc:227] Handling call
0807 08:01:18.356181 (+1467735us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b540021f7d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:18.356334 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1107795200}
I20260807 08:01:18.351881 16397 raft_consensus.cc:3032] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:18.363941 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:18.362320 16397 raft_consensus.cc:729] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:18.362345 13540 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.365267 14583 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:18.338912 16385 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.367213 16377 consensus_queue.cc:237] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.347477 16414 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.338351 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:18.382036 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:18.326807 16440 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.412027 13814 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:01:18.377143 13700 raft_consensus.cc:2379] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:18.421043 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:18.421785 13816 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:18.422643 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.368618 16085 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:18.368697 16176 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:18.383052 16361 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.440424 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:18.454017 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:18.454718 13708 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:18.429342 16425 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.460318 16425 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.388230 16435 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.472401 16435 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:18.473125 16440 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.395031 16193 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.473879 16397 consensus_queue.cc:260] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.476027 16193 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:18.476302 16397 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:18.406059 16385 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:18.491367 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 129110us
I20260807 08:01:18.498133 13769 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.382436 16414 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.501725 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:18.504046 14627 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:18.507875 16414 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.449445 13816 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:18.542315 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.543020 13814 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:18.544910 13537 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:18.515214 13702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1303) took 1460 ms (client timeout 1809 ms). Trace:
I20260807 08:01:18.524942 13804 raft_consensus.cc:1261] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:18.527830 13817 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:18.460383 16105 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.531276 16385 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:18.531615 13588 consensus_meta.cc:287] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.576s user 0.007s sys 0.006s
I20260807 08:01:18.533870 16461 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.533495 16243 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.473161 16435 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.537592 16400 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:18.540381 16422 consensus_meta.cc:287] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.734s user 0.011s sys 0.001s
I20260807 08:01:18.541432 16425 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.541818 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:18.547807 16383 log.cc:924] Time spent T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 1.189s user 0.004s sys 0.001s
I20260807 08:01:18.554064 16401 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.555024 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:18.555330 13590 raft_consensus.cc:1226] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:18.505486 16483 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.556757 13540 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:18.599736 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 44894.3us
I20260807 08:01:18.558550 16465 raft_consensus.cc:3032] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
W20260807 08:01:18.562624 16416 consensus_meta.cc:287] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.617s user 0.011s sys 0.004s
I20260807 08:01:18.509773 16414 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:18.564642 16269 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 08:01:18.572067 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 74025.4us
I20260807 08:01:18.652029 13769 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:18.549212 16315 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:18.578402 16461 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:18.551811 13702 rpcz_store.cc:269] 0807 08:01:17.055170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:17.062215 (+ 7045us) service_pool.cc:227] Handling call
0807 08:01:18.514870 (+1452655us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b54003aacd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:18.515070 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31471488}
I20260807 08:01:18.568661 14619 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:18.574663 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:18.575317 16253 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:18.591941 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:18.674268 13591 raft_consensus.cc:2379] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:18.597654 13579 raft_consensus.cc:2379] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:18.601251 13540 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:18.675324 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 74168.7us
I20260807 08:01:18.584270 16385 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260807 08:01:18.598721 16377 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.653316 13769 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.599669 16401 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.641147 16416 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.603111 16465 raft_consensus.cc:729] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:18.665534 16222 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:18.593592 16422 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:18.660774 16243 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.667531 16105 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.712393 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1309) took 1514 ms. Trace:
I20260807 08:01:18.712774 13705 rpcz_store.cc:276] 0807 08:01:17.198229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:17.198493 (+ 264us) service_pool.cc:227] Handling call
0807 08:01:17.490301 (+291808us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:18.712194 (+1221893us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b5400322654. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:18.712335 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":485964800}
I20260807 08:01:18.715000 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.715781 13814 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:18.686801 16437 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:18.721025 16147 raft_consensus.cc:979] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:18.721309 16437 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:18.722080 16437 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:18.725761 13589 raft_consensus.cc:2379] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 2: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:18.662405 16269 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:18.676016 13540 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:18.576107 13588 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:18.680370 13710 raft_consensus.cc:2379] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 2: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:18.736718 13710 raft_consensus.cc:1226] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 08:01:18.640838 16483 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.602341 16412 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:18.696262 13709 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:18.642613 16216 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.701980 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:18.702376 16401 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.703156 16358 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:18.703500 13703 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:18.668183 16359 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:18.706480 16465 consensus_queue.cc:260] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.708259 16222 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.670581 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:18.753961 13817 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:18.730166 14625 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260807 08:01:18.759269 16172 raft_consensus.cc:2726] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 08:01:18.763332 13711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 235) took 1592 ms (client timeout 1686 ms). Trace:
W20260807 08:01:18.763831 13711 rpcz_store.cc:269] 0807 08:01:17.171109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:17.171242 (+ 133us) service_pool.cc:227] Handling call
0807 08:01:18.763126 (+1591884us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54002b9b54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:18.763264 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":734493056}
I20260807 08:01:18.765241 13817 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:18.769974 16465 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:18.749765 16419 raft_consensus.cc:1067] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Signalling peer 9210dbc4cc9c45e5b26a463b705d362c to start an election
I20260807 08:01:18.705212 16416 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.751741 13537 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:18.751977 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 76023.1us
I20260807 08:01:18.753415 13811 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:18.756232 13769 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:18.742959 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.773545 14580 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:18.772943 16359 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:18.810618 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 58995.2us
I20260807 08:01:18.811308 16412 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:18.784447 16358 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.820112 16459 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:18.676739 16414 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.727483 16377 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.745615 16269 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:18.791397 16429 raft_consensus.cc:2726] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:18.687285 16481 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:18.850283 13700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1641) took 1065 ms. Trace:
I20260807 08:01:18.778823 16216 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.898491 13700 rpcz_store.cc:276] 0807 08:01:17.785105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:17.794386 (+ 9281us) service_pool.cc:227] Handling call
0807 08:01:18.793623 (+999237us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b5400408154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:18.850196 (+ 56573us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1098042240}
I20260807 08:01:18.800175 16478 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:18.852936 14622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 419) took 1721 ms. Trace:
I20260807 08:01:18.595410 16325 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.148s
I20260807 08:01:18.791522 16243 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:18.858209 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 74140.3us
I20260807 08:01:18.927927 14627 raft_consensus.cc:1226] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:18.786389 13769 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.862285 13811 raft_consensus.cc:1226] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:18.862426 13705 raft_consensus.cc:1226] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:18.740823 16313 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.937888 16313 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.938462 16416 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:18.903280 14622 rpcz_store.cc:276] 0807 08:01:17.131635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:17.131797 (+ 162us) service_pool.cc:227] Handling call
0807 08:01:17.132157 (+ 360us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:17.157923 (+ 25766us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:17.157962 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:17.158487 (+ 525us) log.cc:842] Serialized 26 byte log entry
0807 08:01:17.211654 (+ 53167us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:17.362134 (+150480us) log.cc:842] Serialized 14 byte log entry
0807 08:01:17.790106 (+427972us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b18000434e0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b19fba4ca 00007f6b2369bf99 00007f6b19e7e990 00007f6b1a040864 00007f6b1a040155 00007f6b1cf633d0 00007f6b1cf75576 00007f6b1cf860d7 00007f6b1cf86058
0807 08:01:17.802746 (+ 12640us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:17.808864 (+ 6118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:18.546740 (+737876us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:18.852537 (+305797us) raft_consensus.cc:1671] finished
0807 08:01:18.852564 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:18.852813 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50999,"prepare.queue_time_us":22908,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":4397,"raft.queue_time_us":467599,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":19071,"spinlock_wait_cycles":48721408,"thread_start_us":197340,"threads_started":2,"wal-append.queue_time_us":81822}
I20260807 08:01:18.939492 16416 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:18.871955 16144 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260807 08:01:18.873787 16459 raft_consensus.cc:686] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:18.865866 16412 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:18.747293 16461 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.867298 16359 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:18.932801 14627 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
W20260807 08:01:18.944885 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 158553us
I20260807 08:01:18.959395 13769 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.960525 13769 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.951176 16459 consensus_queue.cc:237] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.951328 14619 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:18.954334 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.950225 16401 consensus_queue.cc:1046] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:18.924142 16243 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:18.929564 16222 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.971145 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:18.978152 16400 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.979202 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:18.993898 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:18.995110 16316 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:18.979902 16421 consensus_meta.cc:287] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.636s user 0.008s sys 0.008s
I20260807 08:01:18.979843 16397 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:18.906739 16481 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.000238 16416 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.983613 16358 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:19.002043 16421 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:19.004683 16487 log.cc:924] Time spent T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.157s user 0.000s sys 0.002s
I20260807 08:01:18.947288 16429 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 08:01:18.985245 16393 consensus_meta.cc:287] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.743s user 0.009s sys 0.007s
I20260807 08:01:18.959690 16412 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:18.966408 16313 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:18.987466 16243 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:18.964155 16325 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:18.963999 16015 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:19.027853 16396 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:18.969813 16435 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:19.018205 13807 raft_consensus.cc:1226] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 08:01:19.027130 16414 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:19.027259 16325 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.007256 16216 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:18.986375 16401 consensus_queue.cc:1225] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer 7a889001d6494fe5b7cbbe7da1dae302 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260807 08:01:19.017481 16393 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.041054 14630 raft_consensus.cc:1226] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:19.044852 14583 leader_election.cc:334] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.686s (SENT)
I20260807 08:01:19.045315 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:19.007284 16397 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:18.998773 16253 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:19.007643 16481 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:19.007784 16483 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:19.013585 16377 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:19.122393 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:19.098531 13657 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.011s sys 0.041s
I20260807 08:01:19.055712 16216 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:19.079039 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:19.042601 16393 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:19.080299 16488 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.129544 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:19.078436 16368 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.023s sys 0.052s
I20260807 08:01:19.129624 16396 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:19.130959 16400 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.177913 14629 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260807 08:01:19.131804 16461 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.132285 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:19.083736 16172 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.186823 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:19.189103 13579 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:19.122157 16222 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
W20260807 08:01:19.192664 16315 consensus_meta.cc:287] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.523s user 0.012s sys 0.000s
I20260807 08:01:19.185330 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:19.197242 13814 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
W20260807 08:01:19.138669 13657 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.672s user 0.013s sys 0.042s
I20260807 08:01:19.199769 16172 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:19.056030 16015 consensus_queue.cc:1046] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.129s
I20260807 08:01:19.198868 13537 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:19.173287 16368 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.893s user 0.025s sys 0.055s
W20260807 08:01:19.121552 16254 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.025s sys 0.055s
I20260807 08:01:19.171468 13589 raft_consensus.cc:2379] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:19.051373 16421 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:19.068045 16243 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.182451 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:19.214926 13591 raft_consensus.cc:3037] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:19.133591 16325 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:19.211124 16254 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s user 0.026s sys 0.055s
I20260807 08:01:19.220491 13432 catalog_manager.cc:5495] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 1 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 3 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:19.225546 13432 catalog_manager.cc:5495] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 1 to 3, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 3 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:19.229274 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:19.230103 13811 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
W20260807 08:01:19.241356 13661 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.879s user 0.000s sys 0.020s
W20260807 08:01:19.242794 13661 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.880s user 0.000s sys 0.020s
I20260807 08:01:19.243710 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:19.231381 13430 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57868 (request call id 213) took 3560 ms. Trace:
I20260807 08:01:19.246840 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:19.210086 16377 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:19.213651 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:19.251318 13418 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 0248ade936924e628c46b8de6252af1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 9210dbc4cc9c45e5b26a463b705d362c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:19.253261 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:19.254698 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:19.199362 16461 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.186240 16400 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.214811 16446 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:19.051735 16414 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.260860 13580 raft_consensus.cc:1226] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:19.256367 13809 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.064s user 0.054s sys 0.139s
W20260807 08:01:19.342646 13809 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.150s user 0.055s sys 0.139s
I20260807 08:01:19.268960 14627 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
W20260807 08:01:19.343977 13708 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.874s user 0.019s sys 0.037s
I20260807 08:01:19.280230 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:19.304080 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:19.273166 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:19.269784 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:19.304471 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:19.346158 13703 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 4
I20260807 08:01:19.344806 13811 raft_consensus.cc:2354] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:19.346256 13711 raft_consensus.cc:2396] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260807 08:01:19.348232 13706 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.974s user 0.004s sys 0.032s
W20260807 08:01:19.344273 13708 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.875s user 0.019s sys 0.037s
W20260807 08:01:19.358016 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 47385.3us
W20260807 08:01:19.348678 13706 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.975s user 0.004s sys 0.032s
I20260807 08:01:19.125191 16397 raft_consensus.cc:686] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:19.259588 16316 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:19.221562 16243 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.362922 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:19.044310 16440 consensus_queue.cc:1046] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.295s
I20260807 08:01:19.357508 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:19.366492 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:19.377094 13590 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
W20260807 08:01:19.304109 16485 log.cc:924] Time spent T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260807 08:01:19.269052 16478 raft_consensus.cc:2726] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:19.374346 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:19.347873 14620 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.995s user 0.010s sys 0.026s
I20260807 08:01:19.121968 16429 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.295s
I20260807 08:01:19.348874 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:19.380710 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:19.305279 16361 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.348315 16465 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.343485 16483 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:19.343573 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:19.405061 13814 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:19.411352 14629 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:19.400741 13661 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:19.406073 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:19.345647 16446 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:19.419195 14626 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.004s sys 0.022s
W20260807 08:01:19.419528 14626 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.824s user 0.004s sys 0.022s
I20260807 08:01:19.421665 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:19.424162 13703 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:19.429829 13805 raft_consensus.cc:1226] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:19.408660 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 46039.2us
W20260807 08:01:19.409576 13659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.009s sys 0.013s
I20260807 08:01:19.367364 16285 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:19.438902 13657 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:19.440486 13657 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:19.264357 16377 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.446165 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:19.401489 14620 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.049s user 0.011s sys 0.026s
I20260807 08:01:19.246608 13430 rpcz_store.cc:276] 0807 08:01:15.662357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:15.665707 (+ 3350us) service_pool.cc:227] Handling call
0807 08:01:19.222204 (+3556497us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:17.445891 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:17.476152 (+ 30261us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:17.476203 (+ 51us) write_op.cc:435] Acquired schema lock
0807 08:01:17.476227 (+ 24us) tablet.cc:660] Decoding operations
0807 08:01:17.477031 (+ 804us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:17.477142 (+ 111us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:17.477180 (+ 38us) tablet.cc:683] Acquiring locks for 4 operations
0807 08:01:17.477760 (+ 580us) tablet.cc:699] Row locks acquired
0807 08:01:17.477786 (+ 26us) write_op.cc:260] PREPARE: finished
0807 08:01:17.477967 (+ 181us) write_op.cc:270] Start()
0807 08:01:17.478100 (+ 133us) write_op.cc:276] Timestamp: P: 1786089677477913 usec, L: 0
0807 08:01:17.478128 (+ 28us) op_driver.cc:331] REPLICATION: starting
0807 08:01:17.478918 (+ 790us) log.cc:842] Serialized 1682 byte log entry
0807 08:01:17.480885 (+ 1967us) op_driver.cc:444] REPLICATION: finished
0807 08:01:17.483100 (+ 2215us) write_op.cc:301] APPLY: starting
0807 08:01:17.483178 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:17.483310 (+ 132us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:17.483399 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:17.483926 (+ 527us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:17.484106 (+ 180us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0807 08:01:17.484145 (+ 39us) write_op.cc:312] APPLY: finished
0807 08:01:17.484347 (+ 202us) log.cc:842] Serialized 41 byte log entry
0807 08:01:17.484507 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:17.484594 (+ 87us) write_op.cc:454] Released schema lock
0807 08:01:17.484635 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:17.484864 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:17.484888 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":274,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":1985,"apply.run_cpu_time_us":2031,"apply.run_wall_time_us":2032,"num_ops":4,"prepare.queue_time_us":204317,"prepare.run_cpu_time_us":3828,"prepare.run_wall_time_us":49109,"replication_time_us":2651,"thread_start_us":137971,"threads_started":1}]]}
I20260807 08:01:19.407502 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:19.367120 16393 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:19.448338 13805 raft_consensus.cc:1203] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Deduplicated request from leader. Original: 2.2->[4.3-4.3] Dedup: 4.3->[]
W20260807 08:01:19.431144 13659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.567s user 0.010s sys 0.013s
I20260807 08:01:19.185560 16488 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:19.450488 13816 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.009s sys 0.030s
I20260807 08:01:19.395503 16167 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.397801 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:19.451951 13703 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:19.441383 13711 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:19.409441 16267 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:19.348178 16414 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:19.367656 13701 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.001s sys 0.019s
I20260807 08:01:19.495532 13805 raft_consensus.cc:1226] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 08:01:19.428860 13579 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:01:19.410079 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:19.499667 13816 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s user 0.009s sys 0.030s
W20260807 08:01:19.443403 16455 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.025s user 0.026s sys 0.038s
I20260807 08:01:19.366091 16489 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.145s
I20260807 08:01:19.383304 16316 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.431768 16446 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:19.508103 16224 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.002s sys 0.034s
I20260807 08:01:19.495739 16385 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.491662 16144 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.553617 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:19.431176 16465 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:19.509944 16481 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:19.498355 13588 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 3
I20260807 08:01:19.607995 13705 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
W20260807 08:01:19.543417 13701 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.792s user 0.003s sys 0.020s
W20260807 08:01:19.553416 16147 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.011s sys 0.023s
W20260807 08:01:19.553845 16455 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.140s user 0.027s sys 0.038s
I20260807 08:01:19.610896 13703 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:01:19.603868 16224 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s user 0.002s sys 0.034s
I20260807 08:01:19.452442 16285 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:19.610826 13540 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.027s sys 0.036s
I20260807 08:01:19.429508 16483 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:19.616310 13817 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.028s sys 0.023s
W20260807 08:01:19.754738 13817 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.855s user 0.029s sys 0.028s
W20260807 08:01:19.544879 13808 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.000s sys 0.061s
I20260807 08:01:19.535120 16396 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:19.452200 16435 consensus_queue.cc:1046] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.118s
I20260807 08:01:19.420280 16361 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:19.610155 13702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.021s sys 0.037s
I20260807 08:01:19.507782 16393 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.963513 16393 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:19.964263 16216 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.548942 16440 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:19.964984 16216 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.965932 16440 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:19.967571 16216 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:19.968022 16440 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:19.968222 16216 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:19.671129 13804 consensus_meta.cc:287] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.854s user 0.011s sys 0.011s
I20260807 08:01:19.517365 16167 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:19.741711 13540 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.813s user 0.028s sys 0.036s
W20260807 08:01:20.007972 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 1.07909e+06us
I20260807 08:01:20.009049 13540 leader_election.cc:398] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3.
I20260807 08:01:20.009409 13540 leader_election.cc:401] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260807 08:01:19.641500 16147 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.794s user 0.011s sys 0.023s
I20260807 08:01:19.526343 16267 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:19.638393 16172 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:19.637867 16405 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.027s sys 0.052s
W20260807 08:01:19.645548 16269 consensus_meta.cc:287] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.740s user 0.007s sys 0.007s
W20260807 08:01:19.751834 16417 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.013s sys 0.041s
W20260807 08:01:19.841547 13432 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.004s sys 0.094s
W20260807 08:01:20.018117 13432 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.773s user 0.004s sys 0.094s
I20260807 08:01:19.636462 16385 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:19.804628 16224 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:19.642513 16144 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:19.829629 16459 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.017s sys 0.063s
I20260807 08:01:19.601603 16424 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:19.908109 14583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.011s sys 0.007s
I20260807 08:01:19.973819 16400 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:19.810050 13710 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.016s sys 0.044s
W20260807 08:01:19.909869 13808 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.036s user 0.022s sys 0.079s
I20260807 08:01:19.954573 16435 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:19.955454 16313 raft_consensus.cc:3032] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
W20260807 08:01:19.740651 16368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s user 0.003s sys 0.012s
I20260807 08:01:19.841334 16461 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:19.961581 13702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.042s user 0.028s sys 0.039s
I20260807 08:01:19.975289 13703 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:19.965950 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1252) took 1070 ms. Trace:
I20260807 08:01:20.007680 16440 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:19.843936 16512 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:19.972730 16499 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:19.874256 16489 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:19.842981 16397 consensus_queue.cc:237] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:20.007107 13591 consensus_meta.cc:287] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.789s user 0.001s sys 0.009s
I20260807 08:01:20.068295 13591 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
W20260807 08:01:20.013427 13706 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.001s sys 0.010s
W20260807 08:01:20.013761 13657 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.001s sys 0.022s
W20260807 08:01:20.069748 13657 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.629s user 0.002s sys 0.022s
W20260807 08:01:20.070104 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 629652us
W20260807 08:01:20.069265 13706 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s user 0.002s sys 0.010s
I20260807 08:01:19.968851 16216 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:20.051333 14583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.920s user 0.013s sys 0.013s
I20260807 08:01:19.984302 16243 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:20.052161 13710 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.141s user 0.016s sys 0.050s
I20260807 08:01:20.058645 13817 rpcz_store.cc:276] 0807 08:01:18.895502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.896454 (+ 952us) service_pool.cc:227] Handling call
0807 08:01:18.896862 (+ 408us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.897123 (+ 261us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.897161 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.899739 (+ 2578us) log.cc:842] Serialized 26 byte log entry
0807 08:01:19.810673 (+910934us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:19.810726 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:19.810818 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:19.965710 (+154892us) raft_consensus.cc:1671] finished
0807 08:01:19.965732 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:19.965872 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":855049,"threads_started":1,"wal-append.queue_time_us":1064374}
I20260807 08:01:20.034652 16285 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:20.061007 13661 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.031s sys 0.042s
W20260807 08:01:20.078426 13661 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.666s user 0.031s sys 0.042s
W20260807 08:01:20.021068 16405 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.239s user 0.030s sys 0.052s
W20260807 08:01:20.020792 16417 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.986s user 0.013s sys 0.041s
I20260807 08:01:19.912425 16431 raft_consensus.cc:3032] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
W20260807 08:01:20.051901 16459 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s user 0.017s sys 0.072s
I20260807 08:01:20.012718 13804 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:20.067159 16489 raft_consensus.cc:686] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:20.037093 16269 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.060732 16424 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.027346 16385 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.029168 16267 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.036687 16361 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:20.020447 16193 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.023s sys 0.085s
I20260807 08:01:20.089350 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1250) took 1228 ms. Trace:
W20260807 08:01:20.060405 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 51544.6us
I20260807 08:01:20.144930 13540 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:20.062016 16377 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.061970 16512 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:20.064924 16414 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:20.081272 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 668693us
W20260807 08:01:20.090333 13659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.026s sys 0.040s
W20260807 08:01:20.148115 13659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.651s user 0.027s sys 0.040s
W20260807 08:01:20.090656 14626 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.027s sys 0.029s
W20260807 08:01:20.091233 13537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.001s sys 0.025s
W20260807 08:01:20.136135 16495 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.409s user 0.000s sys 0.001s
I20260807 08:01:20.057497 16313 raft_consensus.cc:729] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:20.120790 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.121436 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
W20260807 08:01:20.090942 13653 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.019s sys 0.026s
W20260807 08:01:20.152496 13653 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.626s user 0.019s sys 0.026s
I20260807 08:01:20.073235 13657 leader_election.cc:398] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:20.153139 13657 leader_election.cc:401] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260807 08:01:20.126960 13584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1599) took 2046 ms (client timeout 1515 ms). Trace:
I20260807 08:01:20.077242 16316 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.061403 16435 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.066179 16440 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.163411 16489 consensus_queue.cc:237] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.164513 16424 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.164605 13703 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260807 08:01:20.135634 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.165771 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:20.165901 13590 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
W20260807 08:01:20.166303 16412 consensus_meta.cc:287] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.137s user 0.009s sys 0.003s
I20260807 08:01:20.166360 13703 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:20.166781 16412 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.167467 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:20.093691 16243 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.055575 16144 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.067413 16499 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.087232 16431 raft_consensus.cc:729] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:20.126047 16172 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.178674 13711 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:20.148537 16512 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:20.148793 13537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.001s sys 0.025s
I20260807 08:01:20.027539 16224 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:20.148665 14626 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.637s user 0.028s sys 0.029s
I20260807 08:01:20.149461 13661 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:20.154263 16167 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:20.153941 13584 rpcz_store.cc:269] 0807 08:01:18.080532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.081616 (+ 1084us) service_pool.cc:227] Handling call
0807 08:01:20.125440 (+2043824us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54003d0ed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:20.126875 (+ 1435us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3734054784}
W20260807 08:01:20.154981 16506 log.cc:924] Time spent T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.165s user 0.001s sys 0.000s
W20260807 08:01:20.155519 16528 log.cc:924] Time spent T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.237s user 0.000s sys 0.001s
I20260807 08:01:20.155406 16216 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:20.156183 16536 log.cc:924] Time spent T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260807 08:01:20.133635 13804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 286) took 1478 ms. Trace:
I20260807 08:01:20.137183 16465 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.160180 16455 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:20.162909 16435 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.143949 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.085474 16488 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.166067 16424 consensus_queue.cc:1046] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:20.168089 13703 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:20.162575 13705 raft_consensus.cc:1261] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:20.142383 16483 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:20.169960 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 96871.3us
W20260807 08:01:20.174371 14580 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.008s sys 0.021s
W20260807 08:01:20.141024 16193 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.009s user 0.023s sys 0.085s
W20260807 08:01:20.223973 14580 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.613s user 0.008s sys 0.022s
I20260807 08:01:20.224092 13657 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:20.224264 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 613460us
I20260807 08:01:20.176934 16459 consensus_queue.cc:1046] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
W20260807 08:01:20.230726 16500 log.cc:924] Time spent T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.387s user 0.007s sys 0.004s
I20260807 08:01:20.232156 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1645) took 2171 ms. Trace:
W20260807 08:01:20.233625 14582 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.008s sys 0.036s
I20260807 08:01:20.201107 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.202019 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.164623 16440 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.206802 13591 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
W20260807 08:01:20.206849 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 57487.9us
W20260807 08:01:20.234946 14582 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.546s user 0.008s sys 0.036s
I20260807 08:01:20.235090 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:20.183157 16497 log.cc:924] Time spent T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20260807 08:01:20.175670 16243 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.171232 16481 raft_consensus.cc:686] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:20.242223 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1640) took 2469 ms. Trace:
I20260807 08:01:20.242339 16552 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:20.215723 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.178505 16396 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:20.179119 16361 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:20.182497 16515 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.169765 16385 consensus_queue.cc:1046] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260807 08:01:20.194921 16414 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.232497 13708 rpcz_store.cc:276] 0807 08:01:18.060808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.465271 (+404463us) service_pool.cc:227] Handling call
0807 08:01:18.465669 (+ 398us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.468799 (+ 3130us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.468829 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.469252 (+ 423us) log.cc:842] Serialized 26 byte log entry
0807 08:01:19.349659 (+880407us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:19.349714 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:19.349824 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.168040 (+818216us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:20.231807 (+ 63767us) raft_consensus.cc:1671] finished
0807 08:01:20.231832 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.232074 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2760,"raft.queue_time_us":74465,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":95260032,"thread_start_us":879893,"threads_started":1,"wal-append.queue_time_us":1168808}
I20260807 08:01:20.211565 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1237) took 2098 ms. Trace:
I20260807 08:01:20.245664 13809 rpcz_store.cc:276] 0807 08:01:18.112811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.112988 (+ 177us) service_pool.cc:227] Handling call
0807 08:01:18.191089 (+ 78101us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b5800004298. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:18.191325 (+ 236us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.191812 (+ 487us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.191843 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.192404 (+ 561us) log.cc:842] Serialized 26 byte log entry
0807 08:01:19.343095 (+1150691us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:19.343132 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:19.343200 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.205167 (+861967us) raft_consensus.cc:1671] finished
0807 08:01:20.205194 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.211485 (+ 6291us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3961,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":131,"spinlock_wait_cycles":50812672,"thread_start_us":1150225,"threads_started":1,"wal-append.queue_time_us":1320145}
I20260807 08:01:20.183350 16540 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:20.214942 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.214460 16316 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:20.195288 16172 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.227051 16544 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.276907 13661 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:20.267823 13708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39134
I20260807 08:01:20.279537 13708 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:20.279985 13708 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:20.283959 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.284832 13580 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:20.267647 13814 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:20.208117 13804 rpcz_store.cc:276] 0807 08:01:18.646299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.660800 (+ 14501us) service_pool.cc:227] Handling call
0807 08:01:20.117087 (+1456287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:20.290834 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.291603 13804 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:20.274885 13703 raft_consensus.cc:1226] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:20.269075 16515 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.294106 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.294658 16515 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.242736 13701 rpcz_store.cc:276] 0807 08:01:17.772383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:17.772525 (+ 142us) service_pool.cc:227] Handling call
0807 08:01:18.748811 (+976286us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00bbce 000000000034aea9 00007f6b21e37a30 00007f6b21efd7b4 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:18.749086 (+ 275us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.749696 (+ 610us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.749808 (+ 112us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.750808 (+ 1000us) log.cc:842] Serialized 26 byte log entry
0807 08:01:19.736227 (+985419us) raft_consensus.cc:1631] Marking committed up to 1
0807 08:01:19.748495 (+ 12268us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:19.809041 (+ 60546us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.236740 (+427699us) raft_consensus.cc:1671] finished
0807 08:01:20.239629 (+ 2889us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.242151 (+ 2522us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":190694,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":142,"spinlock_wait_cycles":6925568,"thread_start_us":890879,"threads_started":1,"wal-append.queue_time_us":972633}
I20260807 08:01:20.148638 13808 rpcz_store.cc:276] 0807 08:01:18.861160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.865607 (+ 4447us) service_pool.cc:227] Handling call
0807 08:01:18.872129 (+ 6522us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.872659 (+ 530us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.872749 (+ 90us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.873784 (+ 1035us) log.cc:842] Serialized 26 byte log entry
0807 08:01:20.061815 (+1188031us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:20.067241 (+ 5426us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:20.073773 (+ 6532us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.076293 (+ 2520us) raft_consensus.cc:1671] finished
0807 08:01:20.079330 (+ 3037us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.087665 (+ 8335us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4378,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":7188,"thread_start_us":1179218,"threads_started":1,"wal-append.queue_time_us":1088601}
I20260807 08:01:20.298826 16243 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.242771 16481 consensus_queue.cc:237] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.300572 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.301211 14625 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:01:20.272850 16539 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:20.303037 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.303786 13589 raft_consensus.cc:2354] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:20.286329 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.287897 16499 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.266767 16361 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.288488 16269 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.288753 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.267736 16529 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:20.331861 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.289366 16547 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260807 08:01:20.304173 16359 consensus_meta.cc:287] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 1.323s user 0.013s sys 0.002s
I20260807 08:01:20.344570 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1254) took 1426 ms. Trace:
I20260807 08:01:20.192760 13711 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:20.306739 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.248214 16412 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.315310 14580 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:20.318305 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.322629 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.328696 16403 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.331067 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:20.333137 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 930688us
I20260807 08:01:20.305585 16316 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:20.309168 16437 consensus_meta.cc:287] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.586s user 0.017s sys 0.002s
I20260807 08:01:20.348627 14630 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 08:01:20.306123 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 102020us
I20260807 08:01:20.364615 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.377782 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.783226 13584 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:20.372514 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:20.439253 16565 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:20.441838 16564 raft_consensus.cc:3032] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:20.841215 16564 raft_consensus.cc:729] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:20.841289 16565 raft_consensus.cc:686] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:20.842077 16564 consensus_queue.cc:260] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.842837 16564 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:20.842687 16565 consensus_queue.cc:237] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.374884 13661 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:20.376534 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:20.380622 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:20.441203 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1329) took 2156 ms. Trace:
I20260807 08:01:20.398298 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:20.885478 13705 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:20.411185 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:20.347756 13700 consensus_meta.cc:287] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.737s user 0.009s sys 0.000s
I20260807 08:01:20.401109 13708 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.369062 13769 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:20.892853 13708 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.348522 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.382177 13808 raft_consensus.cc:3032] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:20.895280 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.895761 13808 raft_consensus.cc:729] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:20.896040 13812 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:20.897071 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.365732 13816 rpcz_store.cc:276] 0807 08:01:18.918700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.918846 (+ 146us) service_pool.cc:227] Handling call
0807 08:01:18.919191 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:18.919434 (+ 243us) raft_consensus.cc:1520] Early marking committed up to index 3
0807 08:01:18.943810 (+ 24376us) log.cc:842] Serialized 14 byte log entry
0807 08:01:20.344208 (+1400398us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:20.344277 (+ 69us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:20.344325 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:20.344395 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.344516 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":798695,"prepare.queue_time_us":24184,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":799379,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":601272,"threads_started":1,"wal-append.queue_time_us":962643}
I20260807 08:01:20.370064 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.898922 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.900035 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.901036 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:20.901566 13816 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:20.370054 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.408629 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.906657 13811 raft_consensus.cc:2396] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 08:01:20.494270 16560 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.908214 16560 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.910826 16560 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:20.357412 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 219000us
W20260807 08:01:20.379454 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 64227.7us
I20260807 08:01:20.344604 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.915678 14580 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:20.347515 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.918818 14623 raft_consensus.cc:2379] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:20.402496 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.920758 14625 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:20.348742 16015 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20260807 08:01:20.924310 14580 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:20.317603 16167 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:20.927023 16167 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.929528 16167 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.929680 13816 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:20.414088 16224 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.420343 16255 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.932220 13657 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:20.932075 16255 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.933308 13809 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:20.934609 16255 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:20.416395 16193 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:20.936210 16193 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.367955 16267 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:20.375741 16316 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:20.371418 16359 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.346199 16368 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:20.945483 13807 raft_consensus.cc:2379] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:20.945734 16368 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.946277 16359 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:20.946664 13807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 295) took 1896 ms (client timeout 2297 ms). Trace:
W20260807 08:01:20.947034 13807 rpcz_store.cc:269] 0807 08:01:19.050719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:19.136076 (+ 85357us) service_pool.cc:227] Handling call
0807 08:01:20.946485 (+1810409us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b540037bed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:20.946612 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4042407296}
I20260807 08:01:20.947739 16368 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:20.332764 16377 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:20.948908 16377 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.950975 16377 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.950948 16316 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.952473 16316 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.266606 16396 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.389099 16397 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:20.445066 16361 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.270356 16414 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.448158 16422 raft_consensus.cc:3032] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:20.377894 16403 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.960894 16403 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:20.304661 16435 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:20.380712 16437 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:20.378487 16446 consensus_meta.cc:287] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.763s user 0.010s sys 0.001s
I20260807 08:01:20.964360 16446 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.964947 16437 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.966511 16446 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.274523 16459 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:20.345992 16496 log.cc:924] Time spent T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.534s user 0.005s sys 0.009s
W20260807 08:01:20.542999 16509 log.cc:924] Time spent T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.812s user 0.001s sys 0.002s
I20260807 08:01:20.465444 16512 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:20.428573 16525 log.cc:924] Time spent T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
I20260807 08:01:20.351842 16529 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.387558 16540 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.310281 16544 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.973060 16529 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:20.348693 16538 log.cc:924] Time spent T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.259s user 0.001s sys 0.001s
I20260807 08:01:20.242908 16552 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.975179 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.975560 16544 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.976790 16552 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.674805 16577 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:20.708511 13588 consensus_meta.cc:287] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.038s user 0.008s sys 0.005s
I20260807 08:01:20.725061 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 243) took 2697 ms. Trace:
I20260807 08:01:20.727106 16412 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:20.977705 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.977808 13588 raft_consensus.cc:1261] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260807 08:01:20.741248 13540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.022s sys 0.054s
W20260807 08:01:20.978477 13540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.816s user 0.023s sys 0.054s
I20260807 08:01:20.977494 16577 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:20.978960 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 816231us
I20260807 08:01:20.977586 13706 rpcz_store.cc:276] 0807 08:01:18.028145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.158369 (+130224us) service_pool.cc:227] Handling call
0807 08:01:18.158756 (+ 387us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.345037 (+186281us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.345069 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.345498 (+ 429us) log.cc:842] Serialized 26 byte log entry
0807 08:01:19.359937 (+1014439us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:19.394765 (+ 34828us) log.cc:842] Serialized 14 byte log entry
0807 08:01:20.071237 (+676472us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:20.724739 (+653502us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b5400407254. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:20.724776 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.724816 (+ 40us) raft_consensus.cc:1671] finished
0807 08:01:20.724889 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.725012 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47789,"prepare.queue_time_us":35654,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":615359,"raft.run_cpu_time_us":437,"raft.run_wall_time_us":81304,"spinlock_wait_cycles":427679488,"thread_start_us":1602633,"threads_started":2,"wal-append.queue_time_us":1521641}
I20260807 08:01:20.980423 16577 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.978040 16412 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.743706 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.758493 13537 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:20.981984 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 223573us
I20260807 08:01:20.982627 16412 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:20.777192 16507 log.cc:924] Time spent T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.723s user 0.000s sys 0.004s
I20260807 08:01:20.782650 13587 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:20.984891 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 621) took 2556 ms. Trace:
I20260807 08:01:20.793231 16417 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.798406 16478 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:20.985841 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:20.986090 16419 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:20.986567 16419 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:20.986349 16478 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.987092 16419 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:20.987287 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.987914 14627 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:20.988359 16478 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.806520 16502 raft_consensus.cc:3032] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:20.988754 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.820730 16254 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:20.989449 16254 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.990377 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.990667 16455 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.991027 14623 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:20.991410 16254 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:20.838657 16313 consensus_queue.cc:260] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.991990 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.823915 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.992559 16313 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:20.992996 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.812141 16172 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.993575 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.993800 16172 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:20.861510 16532 log.cc:924] Time spent T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.728s user 0.000s sys 0.004s
I20260807 08:01:20.760572 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:20.994223 16172 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.994352 16419 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.996124 16172 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.996340 16419 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.996608 16172 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.859323 16596 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:20.877697 13432 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56596 (request call id 307) took 3744 ms. Trace:
I20260807 08:01:20.997349 16596 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.998147 13588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1626) took 1599 ms. Trace:
I20260807 08:01:20.998605 13588 rpcz_store.cc:276] 0807 08:01:19.398395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:19.404611 (+ 6216us) service_pool.cc:227] Handling call
0807 08:01:19.458480 (+ 53869us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:20.997881 (+1539401us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:20.998091 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:20.999466 16596 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:20.860653 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.999686 16424 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 08:01:21.000319 13805 raft_consensus.cc:2379] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:20.879930 16405 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:21.000789 13540 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.001015 16424 consensus_queue.cc:1046] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:21.001196 16401 consensus_queue.cc:1046] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:20.880884 13703 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 1.
I20260807 08:01:21.001891 16424 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.002286 16424 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.002611 16424 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.002686 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.003340 13703 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:21.003561 16269 raft_consensus.cc:3032] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:21.003931 16269 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:21.004096 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.004684 13703 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:20.882954 13702 rpcz_store.cc:276] 0807 08:01:18.285214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.734377 (+449163us) service_pool.cc:227] Handling call
0807 08:01:18.869644 (+135267us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:20.067746 (+1198102us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:20.070242 (+ 2496us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:20.139890 (+ 69648us) log.cc:842] Serialized 26 byte log entry
0807 08:01:20.414898 (+275008us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:20.417816 (+ 2918us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:20.420473 (+ 2657us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.440935 (+ 20462us) raft_consensus.cc:1671] finished
0807 08:01:20.440965 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.441143 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":72594,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":150,"spinlock_wait_cycles":6912,"thread_start_us":1393789,"threads_started":2,"wal-append.queue_time_us":291380}
I20260807 08:01:21.005672 16325 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3.
I20260807 08:01:21.006136 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.006716 16325 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:21.006992 13702 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.008229 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.008548 16325 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:20.890815 13700 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
W20260807 08:01:21.009836 13700 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 246) took 1766 ms (client timeout 1813 ms). Trace:
I20260807 08:01:20.896713 13808 consensus_queue.cc:260] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.010568 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.011054 13808 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.011673 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.012648 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.994223 14622 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.013612 16416 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:20.989374 14627 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:21.014564 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:21.014392 16416 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.015132 14622 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:20.421820 13804 raft_consensus.cc:1261] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 08:01:20.897817 13812 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:21.001031 16405 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.024197 16405 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.024808 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.025383 13817 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:20.997046 16172 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.028498 16172 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.997673 13432 rpcz_store.cc:276] 0807 08:01:17.133750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:17.153990 (+ 20240us) service_pool.cc:227] Handling call
0807 08:01:20.877634 (+3723644us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:20.143839 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:20.156296 (+ 12457us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:20.156336 (+ 40us) write_op.cc:435] Acquired schema lock
0807 08:01:20.156349 (+ 13us) tablet.cc:660] Decoding operations
0807 08:01:20.156792 (+ 443us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:20.156856 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:20.156886 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0807 08:01:20.161928 (+ 5042us) tablet.cc:699] Row locks acquired
0807 08:01:20.161944 (+ 16us) write_op.cc:260] PREPARE: finished
0807 08:01:20.162078 (+ 134us) write_op.cc:270] Start()
0807 08:01:20.162185 (+ 107us) write_op.cc:276] Timestamp: P: 1786089680162038 usec, L: 0
0807 08:01:20.162200 (+ 15us) op_driver.cc:331] REPLICATION: starting
0807 08:01:20.162793 (+ 593us) log.cc:842] Serialized 924 byte log entry
0807 08:01:20.466878 (+304085us) op_driver.cc:444] REPLICATION: finished
0807 08:01:20.468074 (+ 1196us) write_op.cc:301] APPLY: starting
0807 08:01:20.468139 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:20.468251 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:20.468353 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:20.468677 (+ 324us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:20.468789 (+ 112us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0807 08:01:20.468819 (+ 30us) write_op.cc:312] APPLY: finished
0807 08:01:20.468963 (+ 144us) log.cc:842] Serialized 29 byte log entry
0807 08:01:20.469153 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:20.469244 (+ 91us) write_op.cc:454] Released schema lock
0807 08:01:20.469288 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:20.469599 (+ 311us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:20.469625 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":814356,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":936,"apply.run_cpu_time_us":1822,"apply.run_wall_time_us":1830,"num_ops":2,"prepare.queue_time_us":843427,"prepare.run_cpu_time_us":19656,"prepare.run_wall_time_us":200525,"raft.queue_time_us":18462,"raft.run_cpu_time_us":1187,"raft.run_wall_time_us":1187,"replication_time_us":304572,"thread_start_us":946321,"threads_started":4,"wal-append.queue_time_us":249516}]]}
I20260807 08:01:21.030987 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.031742 13805 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:20.897858 16429 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:20.915868 14619 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:20.565815 14618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 623) took 2132 ms. Trace:
I20260807 08:01:20.917366 16393 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:21.091914 16393 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.092567 16393 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:21.092710 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.093564 13812 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:20.925037 16243 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:21.097743 16243 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.098135 16243 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:20.926173 14583 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:21.102082 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 175980us
I20260807 08:01:21.102828 14583 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.103313 13812 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:21.103643 14583 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.104518 16560 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.104696 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.934926 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.105049 16560 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.105463 16529 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:21.105744 16560 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:20.930670 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.107635 13816 raft_consensus.cc:1226] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:20.935096 13657 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:20.939500 16193 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:20.943202 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:21.109565 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 174538us
I20260807 08:01:20.939064 16167 raft_consensus.cc:2781] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:20.954746 16397 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.957320 16414 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.120260 16397 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:20.958331 16422 raft_consensus.cc:729] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:21.122458 16422 consensus_queue.cc:260] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.124241 16422 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:21.124382 16377 consensus_queue.cc:1035] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:21.125016 13657 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:20.963953 13592 raft_consensus.cc:2354] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:21.125834 16414 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.126392 16414 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.126937 16414 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:21.127420 13592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1619) took 1874 ms (client timeout 2115 ms). Trace:
W20260807 08:01:21.127748 13592 rpcz_store.cc:269] 0807 08:01:19.252904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:19.253017 (+ 113us) service_pool.cc:227] Handling call
0807 08:01:21.127246 (+1874229us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b5400204ed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:21.127346 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3922117504}
I20260807 08:01:20.967430 16459 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.130805 14629 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:21.132054 16459 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.132436 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.970738 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1334) took 2646 ms. Trace:
I20260807 08:01:21.134717 16419 raft_consensus.cc:3032] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:20.971997 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1332) took 2648 ms. Trace:
I20260807 08:01:21.135411 16419 raft_consensus.cc:729] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:20.972741 16540 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.134001 13709 rpcz_store.cc:276] 0807 08:01:18.324214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.753819 (+429605us) service_pool.cc:227] Handling call
0807 08:01:18.754100 (+ 281us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.854310 (+100210us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.854342 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.854868 (+ 526us) log.cc:842] Serialized 26 byte log entry
0807 08:01:19.265206 (+410338us) raft_consensus.cc:1631] Marking committed up to 1
0807 08:01:19.265259 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:20.542332 (+1277073us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:20.691093 (+148761us) spinlock_profiling.cc:243] Waited 86.7 ms on lock 0x7b5400081bd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:20.691138 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.970555 (+279417us) raft_consensus.cc:1671] finished
0807 08:01:20.970571 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.970687 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86391,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":155,"spinlock_wait_cycles":222774272,"thread_start_us":409912,"threads_started":1,"wal-append.queue_time_us":690646}
I20260807 08:01:21.135485 13710 rpcz_store.cc:276] 0807 08:01:18.323613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.737595 (+413982us) service_pool.cc:227] Handling call
0807 08:01:18.737941 (+ 346us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.738281 (+ 340us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.738317 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.865095 (+126778us) log.cc:842] Serialized 26 byte log entry
0807 08:01:20.077504 (+1212409us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:20.077578 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:20.077662 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.971789 (+894127us) raft_consensus.cc:1671] finished
0807 08:01:20.971813 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.971933 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":201623,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":185,"thread_start_us":1165793,"threads_started":1,"wal-append.queue_time_us":1336272}
I20260807 08:01:21.136904 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:20.309872 16550 consensus_queue.cc:1046] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.3, Next index: 3, Last known committed idx: 2, Time since last communication: 0.102s
I20260807 08:01:21.140296 16550 consensus_queue.cc:1225] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer 7a889001d6494fe5b7cbbe7da1dae302 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260807 08:01:20.986515 13708 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.142856 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.143633 13708 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:21.144681 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:20.988903 16502 raft_consensus.cc:729] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:21.146513 16502 consensus_queue.cc:260] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.147651 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.148145 16502 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.148408 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.149099 13816 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:21.149133 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:20.866370 16508 log.cc:924] Time spent T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.772s user 0.000s sys 0.005s
I20260807 08:01:21.000001 16465 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:21.153918 16397 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:21.017424 14630 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:21.004773 16269 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.154742 16397 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.156036 16269 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.155436 16431 consensus_queue.cc:260] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.985234 14626 rpcz_store.cc:276] 0807 08:01:18.428500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.428774 (+ 274us) service_pool.cc:227] Handling call
0807 08:01:18.566795 (+138021us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00bb7c 00007f6b1a00b73b 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:18.567020 (+ 225us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.567401 (+ 381us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.567429 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.571656 (+ 4227us) log.cc:842] Serialized 26 byte log entry
0807 08:01:19.419919 (+848263us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:20.185916 (+765997us) log.cc:842] Serialized 14 byte log entry
0807 08:01:20.186108 (+ 192us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:20.186205 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.776696 (+590491us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:20.984707 (+208011us) raft_consensus.cc:1671] finished
0807 08:01:20.984729 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.984839 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":748627,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":56243,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":27980,"spinlock_wait_cycles":55520896,"thread_start_us":1499164,"threads_started":2,"wal-append.queue_time_us":1329321}
I20260807 08:01:20.889609 13701 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:21.159086 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:20.880038 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 505207us
I20260807 08:01:21.160391 13709 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.160662 13661 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:20.893610 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 524606us
I20260807 08:01:21.005470 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:21.038089 13540 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.161911 13703 raft_consensus.cc:2362] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4. Re-sending same reply.
W20260807 08:01:21.162287 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 124259us
I20260807 08:01:21.162463 13769 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:21.162751 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:21.162961 13540 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.163292 13703 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.163928 13769 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:21.163991 16401 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:21.164567 16401 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:21.041209 13817 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.165166 16401 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.165735 16316 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:21.165997 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:20.986124 16417 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.166721 13706 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:21.054913 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.165555 16254 raft_consensus.cc:2726] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:21.170137 13705 raft_consensus.cc:2396] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260807 08:01:21.010353 13700 rpcz_store.cc:269] 0807 08:01:19.243652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:19.248242 (+ 4590us) service_pool.cc:227] Handling call
0807 08:01:21.009760 (+1761518us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:21.170661 13769 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:21.171183 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:21.171975 13705 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.054601 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.173321 13703 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:21.054340 16416 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.062772 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:20.992581 14623 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:21.070509 16424 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.072576 13808 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:21.166743 13817 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:21.075701 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.176477 13811 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.077271 16429 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:21.077924 14622 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.054351 13805 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.079754 16405 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.090488 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:20.346410 16222 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:21.104915 13537 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:21.286440 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.287367 14625 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:21.288295 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.289093 14625 raft_consensus.cc:2379] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:20.954818 16396 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.110842 14583 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.114301 16544 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:21.290422 16544 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:21.291145 16544 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:21.114813 16167 raft_consensus.cc:686] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:21.293466 16167 consensus_queue.cc:237] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:20.969655 14620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 433) took 2638 ms. Trace:
I20260807 08:01:20.971886 16512 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.137540 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:21.136680 16419 consensus_queue.cc:260] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.298395 16419 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.138643 16540 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.136161 16560 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:20.991117 16455 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.301952 16193 raft_consensus.cc:2781] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:21.161319 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:21.164927 16596 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:21.174606 16551 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:21.176959 13661 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.180064 13814 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:21.145370 13708 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:21.304900 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.182422 16552 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:21.186425 13816 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:21.334124 14623 raft_consensus.cc:1226] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:21.186830 13540 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.336099 16540 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:21.187911 13706 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260807 08:01:21.191039 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.169219 16417 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:21.174415 14619 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.165154 16446 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:21.199436 13809 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:21.201982 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.210446 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:21.215080 13769 connection.cc:232] RPC call timeout handler was delayed by -2.3656220859847963s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
I20260807 08:01:21.230188 13811 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.186250 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.376494 13808 raft_consensus.cc:2396] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 08:01:21.233007 13711 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:21.193051 16478 raft_consensus.cc:2741] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260807 08:01:21.240181 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.240156 13705 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.248162 13805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "76bbc935aada4ceab2774e363186745d"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:21.380686 13805 raft_consensus.cc:595] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:21.380874 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.212852 16397 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:21.231060 16015 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:21.382318 16015 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.198786 16316 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260807 08:01:21.226331 16254 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:21.385054 13769 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:21.387216 16596 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.206648 16243 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.387588 16596 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:21.388036 16596 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.249637 16424 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.228338 16393 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.391819 16403 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:21.161592 14618 rpcz_store.cc:276] 0807 08:01:18.433219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.434482 (+ 1263us) service_pool.cc:227] Handling call
0807 08:01:19.041144 (+606662us) spinlock_profiling.cc:243] Waited 5.55 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:19.041448 (+ 304us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:19.042033 (+ 585us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:19.042115 (+ 82us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:19.042804 (+ 689us) log.cc:842] Serialized 26 byte log entry
0807 08:01:19.514139 (+471335us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:20.140591 (+626452us) log.cc:842] Serialized 14 byte log entry
0807 08:01:20.565325 (+424734us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b54006822a0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cefefb3 00007f6b1cf6bf0f 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c
0807 08:01:20.565378 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:20.565461 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.565498 (+ 37us) raft_consensus.cc:1671] finished
0807 08:01:20.565564 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.565753 (+ 189us) inbound_call.cc:173] Queueing success response
0807 08:01:20.834656 (+268903us) spinlock_profiling.cc:243] Waited 8.18 ms on lock 0x7b18003b0080. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043764 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
Metrics: {"prepare.queue_time_us":678073,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":5582,"raft.queue_time_us":66212,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":78896,"spinlock_wait_cycles":144702720,"thread_start_us":1108528,"threads_started":3,"wal-append.queue_time_us":766290}
I20260807 08:01:21.284529 16222 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.305472 13708 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
W20260807 08:01:21.339370 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 152639us
I20260807 08:01:21.400112 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.400523 13540 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.401789 13540 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:21.339501 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 234725us
I20260807 08:01:21.233724 13657 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.446178 16243 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:21.342682 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 229896us
I20260807 08:01:21.301784 16431 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:21.302783 16193 raft_consensus.cc:686] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:21.369801 14629 raft_consensus.cc:1226] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:21.381724 13705 raft_consensus.cc:2379] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:21.471871 14583 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:21.385870 16254 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.254896 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:21.439671 16172 consensus_queue.cc:1046] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260807 08:01:21.344664 16551 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.483729 16551 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.484169 16551 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.487619 16401 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.487856 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.488631 13702 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:21.489631 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.369096 16566 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:21.490420 16566 raft_consensus.cc:686] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:21.491431 16566 consensus_queue.cc:237] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.369572 16405 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.394915 16446 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.493985 16446 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.494383 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.494580 16414 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.496518 16414 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.423818 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.497841 13710 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
W20260807 08:01:21.426337 16621 log.cc:924] Time spent T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20260807 08:01:21.426906 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.498684 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.499434 13591 raft_consensus.cc:2379] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:21.499146 14625 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:21.499210 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:21.473906 16455 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.500268 13589 raft_consensus.cc:2379] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:21.474068 16316 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:21.428444 13540 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:21.501803 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 73445.6us
I20260807 08:01:21.502050 16269 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:21.502418 16316 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.503038 16316 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:21.503149 13540 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.503952 16254 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:21.504215 13540 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:21.398712 16393 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.505424 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.430459 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.430526 13537 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:21.385070 16015 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.392716 16403 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.413509 16412 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.549270 16412 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.549964 16412 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.551517 16403 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.442437 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.555162 13706 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:21.420763 16560 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:21.555852 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 125449us
I20260807 08:01:21.432818 16224 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:21.556774 16224 raft_consensus.cc:686] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:21.557034 16455 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:21.557973 16455 raft_consensus.cc:686] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:21.558687 16224 consensus_queue.cc:237] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.559617 16455 consensus_queue.cc:237] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.459355 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.561825 13805 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:21.564209 13589 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260807 08:01:21.340907 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 163994us
I20260807 08:01:21.565585 16414 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:21.566069 13661 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.566435 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:21.566395 16414 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.567232 13589 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:21.567925 13661 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.568882 16540 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:21.296367 14620 rpcz_store.cc:276] 0807 08:01:18.331535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:18.338148 (+ 6613us) service_pool.cc:227] Handling call
0807 08:01:18.350236 (+ 12088us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:18.351182 (+ 946us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:18.351264 (+ 82us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:18.352092 (+ 828us) log.cc:842] Serialized 26 byte log entry
0807 08:01:19.510750 (+1158658us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:19.516325 (+ 5575us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:19.541937 (+ 25612us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:20.345344 (+803407us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:20.969367 (+624023us) raft_consensus.cc:1671] finished
0807 08:01:20.969384 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:20.969605 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2490,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":12524,"spinlock_wait_cycles":4953600,"thread_start_us":1136940,"threads_started":1,"wal-append.queue_time_us":1264280}
I20260807 08:01:21.569370 16540 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:21.569165 16405 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:21.569895 16361 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:21.570295 16540 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:21.572266 16414 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.573787 16437 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 08:01:21.466176 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 232560us
I20260807 08:01:21.412285 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.475545 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.479534 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.479684 16397 consensus_queue.cc:1046] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:21.576236 16397 raft_consensus.cc:3032] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:21.576696 16397 raft_consensus.cc:729] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:21.577907 16397 consensus_queue.cc:260] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.481225 16222 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.481125 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.482065 16193 consensus_queue.cc:237] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.483191 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.416052 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 251) took 1043 ms. Trace:
I20260807 08:01:21.512806 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:21.594694 13711 rpcz_store.cc:276] 0807 08:01:20.372553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:20.380377 (+ 7824us) service_pool.cc:227] Handling call
0807 08:01:21.415813 (+1035436us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b54001dbad4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:21.415981 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":789989760}
W20260807 08:01:21.479190 14583 connection.cc:232] RPC call timeout handler was delayed by -1.3651288729161024s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18130000000000002s for a response
W20260807 08:01:21.596076 14583 connection.cc:232] RPC call timeout handler was delayed by -1.2154968827962875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
I20260807 08:01:21.500170 16243 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.496951 16596 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.527307 16401 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.528788 16393 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:21.524089 13432 catalog_manager.cc:5495] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:21.529700 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.542510 16172 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.529938 16254 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:21.534250 13703 raft_consensus.cc:1226] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:21.528297 13540 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.540971 16551 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.548422 16424 raft_consensus.cc:3032] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:21.414484 16577 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:21.454020 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.631794 13816 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:21.581698 13708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8727dbda122418487cb692370c2399e"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:59228
W20260807 08:01:21.630383 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 102181us
I20260807 08:01:21.586505 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.587723 13589 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
W20260807 08:01:21.633880 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 64396.3us
I20260807 08:01:21.592461 13805 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:21.595974 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.505230 16269 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:21.633241 13708 raft_consensus.cc:595] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.636009 13708 raft_consensus.cc:603] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:21.635565 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.637529 13711 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:21.597955 14583 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:21.636754 14580 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:21.639969 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:21.599265 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.599608 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.599268 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.603724 16243 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.602195 16544 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.606187 16596 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.606300 16446 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.612962 13432 catalog_manager.cc:5495] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 2 to 3, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:21.657786 13432 catalog_manager.cc:5495] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:21.598397 13657 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.631438 16424 raft_consensus.cc:729] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:21.625913 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.632478 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:21.663849 13703 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:21.663180 13432 catalog_manager.cc:5495] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 2 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:21.665205 13703 raft_consensus.cc:1226] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:21.604281 16412 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:21.582228 13849 debug-util.cc:398] Leaking SignalData structure 0x7b0800058f80 after lost signal to thread 11828
W20260807 08:01:21.642050 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 85363.4us
I20260807 08:01:21.634943 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.677845 13592 raft_consensus.cc:2379] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:21.644204 13817 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:21.626041 16397 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.674817 13432 catalog_manager.cc:5495] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:21.648159 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:21.631340 16405 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:21.681882 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.681948 16405 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.683760 16552 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:21.684075 16405 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.626451 16172 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.626170 16633 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.635231 16401 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
W20260807 08:01:21.659893 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 61601.7us
I20260807 08:01:21.685251 13711 raft_consensus.cc:1261] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260807 08:01:21.611224 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 98515.6us
I20260807 08:01:21.686841 13769 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.687247 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.641330 16560 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:21.641688 16361 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:21.668745 13709 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:21.653461 16502 consensus_meta.cc:287] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.505s user 0.020s sys 0.008s
I20260807 08:01:21.648324 16243 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.692236 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.685389 13657 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:21.693316 13584 raft_consensus.cc:1226] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:21.689373 13537 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:21.696962 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:21.684324 16552 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:21.643823 16313 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260807 08:01:21.712565 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:21.639052 16577 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.652752 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.689337 13769 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:21.656558 16565 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.656586 16403 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:21.734736 16403 raft_consensus.cc:686] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:21.734786 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.735538 14619 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:21.735713 16403 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.656608 16269 raft_consensus.cc:686] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:21.631912 16254 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.698032 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:21.738315 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 49045.1us
I20260807 08:01:21.723276 13804 raft_consensus.cc:2396] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:21.687970 13591 raft_consensus.cc:2396] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:21.742331 13579 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:21.742506 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.742888 16269 consensus_queue.cc:237] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.701265 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.710507 13709 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:21.711615 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:21.744246 14626 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.744776 16577 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.746088 16465 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:21.746510 16465 raft_consensus.cc:686] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:21.640164 13540 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.747476 16254 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:21.748147 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 108053us
I20260807 08:01:21.748607 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.670719 16419 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:21.749280 13540 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.749924 16419 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.750654 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:21.752872 16419 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.756392 16419 raft_consensus.cc:2726] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:21.724934 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.686257 16633 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.729347 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.760186 13584 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:21.685999 16172 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.734237 16565 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.701946 16222 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.690248 16416 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:21.677711 13816 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:21.700412 16405 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:21.712224 16552 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.696918 16361 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:21.667025 16396 raft_consensus.cc:2741] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260807 08:01:21.753831 13590 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 08:01:21.757014 14626 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:21.757715 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.758415 16412 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:21.713347 13592 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:21.759917 14630 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:21.629127 16393 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:21.701481 13418 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 0248ade936924e628c46b8de6252af1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 9210dbc4cc9c45e5b26a463b705d362c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:21.764058 14580 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:21.775162 13418 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 73751.6us
I20260807 08:01:21.723028 13701 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:21.737313 13700 raft_consensus.cc:1226] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:21.740272 13849 debug-util.cc:398] Leaking SignalData structure 0x7b08001ddb40 after lost signal to thread 13481
I20260807 08:01:21.730198 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.782469 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:21.785297 13769 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:21.729950 16424 consensus_queue.cc:260] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:21.763430 16015 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:21.787012 16596 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:21.771670 14625 raft_consensus.cc:2354] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:21.790009 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.762030 16565 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.792071 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:21.762310 16222 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:21.762104 16633 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.765059 16416 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:21.816704 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.768219 16361 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.818303 14629 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:21.769011 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:21.827610 13849 debug-util.cc:398] Leaking SignalData structure 0x7b08003bf340 after lost signal to thread 13630
I20260807 08:01:21.776155 16405 raft_consensus.cc:686] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:21.747596 16465 consensus_queue.cc:237] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:21.851112 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 167849us
I20260807 08:01:21.833870 16254 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:21.837694 13592 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:21.790372 16412 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:21.787012 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.790818 16313 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 08:01:21.887914 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 187243us
I20260807 08:01:21.839437 13590 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:21.819293 16015 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:21.845710 16414 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:21.834878 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:21.835481 16551 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.838747 16424 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.843254 16222 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.908888 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:21.897261 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 207952us
I20260807 08:01:21.841547 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:21.843122 16416 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.842949 16560 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:21.903761 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:21.881687 16544 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:21.917666 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.913126 13661 leader_election.cc:398] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:21.914001 13814 raft_consensus.cc:1261] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:21.883054 16417 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:21.883417 16412 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.897054 16313 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:21.900051 16446 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:21.920223 13849 debug-util.cc:398] Leaking SignalData structure 0x7b080021f280 after lost signal to thread 13743
I20260807 08:01:21.924458 13814 raft_consensus.cc:1226] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:21.900425 16254 raft_consensus.cc:686] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:21.905318 16015 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:21.929142 13817 raft_consensus.cc:1261] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 08:01:21.923462 13661 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:21.940151 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.943454 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:21.907938 16401 raft_consensus.cc:686] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:21.953152 13814 raft_consensus.cc:2354] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:21.922971 16596 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:21.960063 13814 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc32413495ee41909c7c5494c8095084"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:21.925302 16560 raft_consensus.cc:686] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:21.932631 16412 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.935527 16313 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:21.911442 16414 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:21.970443 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 219858us
I20260807 08:01:21.959765 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:21.991487 13540 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:21.966280 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:21.985846 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 131560us
I20260807 08:01:21.933629 16417 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:21.978830 16502 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:21.933351 16405 consensus_queue.cc:237] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:22.012764 14583 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:22.004179 16437 raft_consensus.cc:2726] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 08:01:22.028093 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 104734us
I20260807 08:01:22.029736 13661 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:22.031673 13709 raft_consensus.cc:1226] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 08:01:22.034502 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:22.035063 16544 consensus_queue.cc:1035] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:22.035740 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:22.010803 16393 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:22.011111 16502 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:22.041031 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:22.056455 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:22.057158 13711 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:22.041253 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:22.060019 13703 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:22.047106 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:22.066689 14627 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:22.072182 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:22.072916 13705 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:22.062554 13804 raft_consensus.cc:2396] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260807 08:01:22.089756 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 144593us
I20260807 08:01:22.066205 16502 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:22.085877 16540 consensus_meta.cc:287] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.515s user 0.006s sys 0.007s
I20260807 08:01:22.110419 13711 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
W20260807 08:01:22.163816 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 149320us
I20260807 08:01:22.112353 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:22.010872 16254 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.074587 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1368) took 1036 ms. Trace:
I20260807 08:01:22.168751 13703 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:22.017057 16401 consensus_queue.cc:237] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:22.111205 14622 raft_consensus.cc:1226] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:22.026762 16560 consensus_queue.cc:237] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:22.197685 16636 log.cc:924] Time spent T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.278s user 0.000s sys 0.001s
W20260807 08:01:22.226122 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 196485us
I20260807 08:01:22.110847 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:22.167275 16437 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:22.167027 16678 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:22.173151 14623 raft_consensus.cc:1226] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260807 08:01:22.216017 16643 log.cc:924] Time spent T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
W20260807 08:01:22.207487 16532 log.cc:924] Time spent T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260807 08:01:22.245179 16673 log.cc:924] Time spent T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.208s user 0.000s sys 0.001s
I20260807 08:01:22.246677 16512 consensus_queue.cc:1035] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:22.253487 13661 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:22.227639 16637 log.cc:924] Time spent T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.293s user 0.000s sys 0.001s
I20260807 08:01:22.270391 13661 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:22.271858 13661 leader_election.cc:398] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:22.275466 13661 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:22.286723 16639 log.cc:924] Time spent T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.282s user 0.000s sys 0.001s
W20260807 08:01:22.287019 16644 log.cc:924] Time spent T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.299s user 0.000s sys 0.001s
I20260807 08:01:22.174916 16540 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:22.261260 16646 log.cc:924] Time spent T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.272s user 0.000s sys 0.001s
I20260807 08:01:22.293416 13661 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:22.243281 16649 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:22.297616 13661 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:22.251358 16697 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 08:01:22.301749 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 191285us
W20260807 08:01:22.288934 16577 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.013s sys 0.023s
I20260807 08:01:22.203729 13701 rpcz_store.cc:276] 0807 08:01:21.037707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.158647 (+120940us) service_pool.cc:227] Handling call
0807 08:01:22.046209 (+887562us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54003f0554. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:22.063467 (+ 17258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1243264768}
I20260807 08:01:22.173961 16172 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.300930 13661 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:22.258351 16437 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:22.326027 16316 consensus_meta.cc:287] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.822s user 0.011s sys 0.002s
W20260807 08:01:22.323567 16577 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.564s user 0.013s sys 0.024s
I20260807 08:01:22.333956 13657 leader_election.cc:398] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:22.390138 13657 leader_election.cc:401] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260807 08:01:22.324558 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1367) took 1274 ms. Trace:
I20260807 08:01:22.327677 16417 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:22.306392 16679 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:22.296702 16633 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.333168 16697 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:22.301162 16661 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:22.414726 16669 raft_consensus.cc:2781] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:22.420657 13811 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:22.424489 16688 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:22.304183 13812 raft_consensus.cc:1261] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260807 08:01:22.426357 16424 consensus_meta.cc:287] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.516s user 0.010s sys 0.005s
I20260807 08:01:22.406702 13702 rpcz_store.cc:276] 0807 08:01:21.034739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.490621 (+455882us) service_pool.cc:227] Handling call
0807 08:01:21.490990 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:21.497877 (+ 6887us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:21.497908 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:21.498379 (+ 471us) log.cc:842] Serialized 26 byte log entry
0807 08:01:21.819629 (+321250us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:21.883499 (+ 63870us) log.cc:842] Serialized 14 byte log entry
0807 08:01:21.905966 (+ 22467us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:21.911797 (+ 5831us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:22.292152 (+380355us) raft_consensus.cc:1671] finished
0807 08:01:22.292177 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:22.308621 (+ 16444us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":17509,"prepare.run_wall_time_us":43,"raft.queue_time_us":11582,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"spinlock_wait_cycles":19352320,"thread_start_us":294079,"threads_started":1,"wal-append.queue_time_us":399102}
I20260807 08:01:22.426857 16679 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.435238 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:22.437027 13769 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.001s sys 0.009s
W20260807 08:01:22.435474 16416 consensus_meta.cc:287] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.510s user 0.003s sys 0.011s
W20260807 08:01:22.453698 13769 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.544s user 0.001s sys 0.010s
W20260807 08:01:22.454147 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 545294us
W20260807 08:01:22.436442 16222 consensus_meta.cc:287] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.516s user 0.005s sys 0.005s
W20260807 08:01:22.437633 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 426746us
I20260807 08:01:22.308509 16649 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:22.470180 14583 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:22.471400 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 132275us
I20260807 08:01:22.476262 13769 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:22.436162 16697 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:22.401371 16316 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:22.482236 14583 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:22.483726 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:22.484356 13708 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:22.485144 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:22.485621 13769 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:22.487891 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:22.440685 16669 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:22.490051 14583 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:22.503172 13811 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:22.447187 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:22.498596 13708 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:22.446452 16661 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 08:01:22.476186 16269 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.011s sys 0.029s
W20260807 08:01:22.507097 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 173294us
I20260807 08:01:22.405478 16686 consensus_queue.cc:1035] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.109s
I20260807 08:01:22.511442 13703 raft_consensus.cc:1226] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:22.447387 16465 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
W20260807 08:01:22.510634 16269 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.011s sys 0.029s
I20260807 08:01:22.494252 16417 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:22.494472 16678 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:22.467835 16416 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.532842 14583 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:22.538897 16681 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 08:01:22.529804 16243 consensus_meta.cc:287] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.807s user 0.011s sys 0.002s
I20260807 08:01:22.488983 16697 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:22.524317 16359 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.051s
I20260807 08:01:22.504570 16669 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:22.494668 16596 consensus_queue.cc:1046] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260807 08:01:22.578589 16703 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3.
I20260807 08:01:22.579335 16316 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:22.519438 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:22.582175 13701 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 08:01:22.581985 16243 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.587173 16243 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:22.589699 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 76881us
I20260807 08:01:22.474944 16222 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:22.494668 16694 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:22.573047 16446 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:22.573385 14627 consensus_meta.cc:287] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.506s user 0.010s sys 0.001s
I20260807 08:01:22.545254 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:22.532527 16224 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
W20260807 08:01:22.561169 16692 log.cc:924] Time spent T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260807 08:01:22.574163 16688 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:22.582559 16633 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:22.584266 16717 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:22.503551 16478 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:22.503674 16459 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20260807 08:01:22.606796 13711 raft_consensus.cc:1226] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:22.478219 16702 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:22.588621 16540 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:22.579308 16703 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:22.590021 16649 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:22.591529 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:22.653218 13809 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:22.557324 13585 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:22.660403 13537 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:22.662111 13807 raft_consensus.cc:1226] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
W20260807 08:01:22.668809 16705 log.cc:924] Time spent T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260807 08:01:22.595821 14583 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.574s (SENT) [suppressed 10 similar messages]
W20260807 08:01:22.680224 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 84422.5us
I20260807 08:01:22.548062 13811 raft_consensus.cc:1261] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:01:22.595989 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:22.683106 13580 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:22.596896 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:22.531765 16661 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:22.598979 14627 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:22.684279 13592 raft_consensus.cc:2379] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
W20260807 08:01:22.574240 16695 log.cc:924] Time spent T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.172s user 0.000s sys 0.001s
I20260807 08:01:22.559275 16417 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:22.687191 16661 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:22.732391 13809 raft_consensus.cc:1261] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260807 08:01:22.703568 16361 consensus_meta.cc:287] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.853s user 0.011s sys 0.000s
I20260807 08:01:22.703866 16269 consensus_queue.cc:1035] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:22.746755 13814 raft_consensus.cc:595] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
W20260807 08:01:22.747797 16412 consensus_meta.cc:287] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.765s user 0.008s sys 0.003s
I20260807 08:01:22.747929 16661 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:22.710590 16718 raft_consensus.cc:3032] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
W20260807 08:01:22.748731 16565 consensus_meta.cc:287] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.900s user 0.009s sys 0.003s
I20260807 08:01:22.748319 16412 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.748862 16718 raft_consensus.cc:729] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:22.715166 16222 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:22.749289 16565 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.750947 14630 raft_consensus.cc:1226] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260807 08:01:22.750325 16718 consensus_queue.cc:260] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.752261 16718 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:22.710929 13811 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 08:01:22.754612 13811 pending_rounds.cc:76] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 5
I20260807 08:01:22.755245 13811 pending_rounds.cc:97] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260807 08:01:22.755703 13811 raft_consensus.cc:2866] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 08:01:22.712484 16015 consensus_meta.cc:287] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.746s user 0.005s sys 0.005s
I20260807 08:01:22.715164 16669 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:22.687763 16649 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:22.663074 16424 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:22.814402 16681 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:22.815420 16424 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:22.750561 16361 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.818586 16669 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:22.820909 16361 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:22.821763 16269 consensus_queue.cc:1035] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:22.821987 16661 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.823599 16424 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:22.814656 16649 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:22.826126 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:22.811483 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 54428.2us
I20260807 08:01:22.886452 14583 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:22.667474 16172 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:22.826529 16015 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.752339 16565 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:22.907524 13592 raft_consensus.cc:1261] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 08:01:22.755159 16417 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:22.603539 16446 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.864660 13805 raft_consensus.cc:1226] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:22.912042 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:22.878651 16709 log.cc:924] Time spent T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.284s user 0.000s sys 0.001s
I20260807 08:01:22.855973 16437 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:22.881587 16679 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:22.886653 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:22.896184 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:22.607431 16688 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.909387 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:01:22.911950 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 403577us
W20260807 08:01:22.786885 16313 consensus_meta.cc:287] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.802s user 0.006s sys 0.006s
I20260807 08:01:22.892359 16649 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:22.914745 13706 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 6
I20260807 08:01:22.914819 13816 raft_consensus.cc:1226] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:22.904624 16678 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:22.916693 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:22.925853 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:22.944650 13709 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:22.945276 13592 raft_consensus.cc:2396] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 08:01:22.906564 16681 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:22.942040 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:22.946671 13809 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:22.947853 14583 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:22.948076 14580 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:22.927480 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:22.958775 16243 consensus_queue.cc:1035] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260807 08:01:22.959720 16641 log.cc:924] Time spent T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.938s user 0.007s sys 0.002s
I20260807 08:01:22.960623 16243 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 08:01:22.970351 16635 log.cc:924] Time spent T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 1.120s user 0.005s sys 0.009s
I20260807 08:01:22.970575 16015 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:22.971889 16405 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:22.972234 16446 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:22.972069 13661 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:22.972446 16405 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:22.972807 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:22.973428 13702 raft_consensus.cc:2362] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2. Re-sending same reply.
I20260807 08:01:22.973577 13661 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:22.974627 13661 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:22.974522 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:22.974850 16405 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:22.975507 16405 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:22.976176 16405 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:22.976624 16361 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:22.977284 16361 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:22.977629 13579 raft_consensus.cc:1261] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:22.977843 16361 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:22.979130 13661 leader_election.cc:398] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:22.979533 13661 leader_election.cc:401] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:22.979329 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:22.976644 16405 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:22.982534 16405 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:22.980610 13579 raft_consensus.cc:2354] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:22.981769 13661 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:23.017876 13706 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:23.018399 13709 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:23.018904 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:23.019273 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:23.019824 13706 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:23.018383 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:23.021306 13706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:01:23.021677 13580 pending_rounds.cc:76] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Aborting all ops after (but not including) 1
I20260807 08:01:23.036222 13580 pending_rounds.cc:97] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260807 08:01:23.036805 13580 raft_consensus.cc:2866] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:22.943408 16679 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:23.039671 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:22.940853 16416 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:23.042979 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:23.040308 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 447334us
I20260807 08:01:22.945083 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:23.056974 13590 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:23.024099 16015 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:23.035993 16502 consensus_meta.cc:287] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.860s user 0.013s sys 0.000s
I20260807 08:01:23.039778 16683 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:23.040053 16686 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:23.023583 16565 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:23.065778 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:22.951018 16313 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.986413 16681 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:22.976183 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:23.052112 16679 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:23.022193 13706 raft_consensus.cc:595] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:22.994671 16577 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.004s sys 0.005s
W20260807 08:01:23.049567 16552 consensus_meta.cc:287] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 1.266s user 0.009s sys 0.008s
I20260807 08:01:23.056149 16722 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
W20260807 08:01:23.115043 14579 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.006s sys 0.011s
W20260807 08:01:23.176265 14579 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.006s sys 0.011s
I20260807 08:01:23.178330 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1643) took 1485 ms. Trace:
I20260807 08:01:23.178795 13585 rpcz_store.cc:276] 0807 08:01:21.692733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.693600 (+ 867us) service_pool.cc:227] Handling call
0807 08:01:23.178121 (+1484521us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b5400369a54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:23.178245 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1771587456}
I20260807 08:01:23.058593 16660 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:23.085974 16746 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:23.110335 16254 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:23.160815 16412 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:23.157651 13701 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:23.162528 13588 raft_consensus.cc:2379] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:23.162801 16313 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:23.163029 16222 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:23.164638 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:23.165874 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:23.166447 16679 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:23.110342 16565 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:23.110774 16633 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:23.167833 16684 log.cc:924] Time spent T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.951s user 0.002s sys 0.012s
I20260807 08:01:23.111198 16683 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:23.157677 13706 raft_consensus.cc:603] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:23.109830 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:23.169157 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:23.113642 16359 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:23.133158 13700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1361) took 2088 ms. Trace:
I20260807 08:01:23.078694 16502 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:23.175185 14629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 443) took 1474 ms (client timeout 1564 ms). Trace:
I20260807 08:01:23.174474 16552 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:23.117125 16661 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:23.183393 16681 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:23.180747 16577 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s user 0.004s sys 0.005s
I20260807 08:01:23.161655 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:23.114019 16478 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:23.076500 16015 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:23.188313 16746 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:23.194466 14622 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.004s sys 0.029s
I20260807 08:01:23.207412 13661 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:23.236784 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 459) took 1228 ms. Trace:
I20260807 08:01:23.663085 14630 rpcz_store.cc:276] 0807 08:01:22.008172 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.008333 (+ 161us) service_pool.cc:227] Handling call
0807 08:01:22.008674 (+ 341us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:22.751402 (+742728us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:23.236409 (+485007us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54006d0454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:23.236686 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1706446592}
I20260807 08:01:23.338507 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1293) took 1423 ms. Trace:
W20260807 08:01:23.629096 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 681302us
I20260807 08:01:23.668877 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:23.656956 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 464386us
I20260807 08:01:23.695927 13709 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:23.209625 13590 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
W20260807 08:01:23.411804 13811 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.006s sys 0.014s
W20260807 08:01:23.704547 13811 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.816s user 0.015s sys 0.015s
W20260807 08:01:23.718340 14629 rpcz_store.cc:269] 0807 08:01:21.700903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.701058 (+ 155us) service_pool.cc:227] Handling call
0807 08:01:23.174943 (+1473885us) spinlock_profiling.cc:243] Waited 74.3 ms on lock 0x7b54006045d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:23.175070 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":170898176}
I20260807 08:01:23.724728 16723 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:23.666097 13807 rpcz_store.cc:276] 0807 08:01:21.915512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.915657 (+ 145us) service_pool.cc:227] Handling call
0807 08:01:21.921756 (+ 6099us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:22.662636 (+740880us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:23.338194 (+675558us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b540006b654. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:23.338428 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1681019776}
I20260807 08:01:23.726460 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:23.727298 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 58596.4us
I20260807 08:01:23.728880 14583 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:23.751768 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:23.751993 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 78936.5us
I20260807 08:01:23.753768 13661 leader_election.cc:398] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:23.754411 13661 leader_election.cc:401] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260807 08:01:23.630995 13584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 326) took 1036 ms. Trace:
I20260807 08:01:23.332775 16596 raft_consensus.cc:979] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:01:23.200582 16686 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:23.224582 13702 raft_consensus.cc:2379] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
W20260807 08:01:23.233714 16697 consensus_meta.cc:287] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.659s user 0.011s sys 0.000s
I20260807 08:01:23.766512 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:23.788606 13706 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:23.232654 16688 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:23.788980 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:23.771817 13709 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:23.756353 13584 rpcz_store.cc:276] 0807 08:01:22.594987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.596443 (+ 1456us) service_pool.cc:227] Handling call
0807 08:01:22.608005 (+ 11562us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:22.608236 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:22.608269 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:22.651566 (+ 43297us) log.cc:842] Serialized 26 byte log entry
0807 08:01:23.073750 (+422184us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:23.073809 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:23.073903 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:23.630686 (+556783us) raft_consensus.cc:1671] finished
0807 08:01:23.630712 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:23.630897 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":319503,"threads_started":1,"wal-append.queue_time_us":951899}
I20260807 08:01:23.775424 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:23.799961 13590 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:23.800089 13657 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:23.337265 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:23.618978 16698 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:23.783134 13705 raft_consensus.cc:2379] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:23.224349 16565 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:23.224179 16313 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:23.814977 16396 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:23.814849 16313 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:23.815716 16396 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:23.818408 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1294) took 1902 ms. Trace:
I20260807 08:01:23.818935 13805 rpcz_store.cc:276] 0807 08:01:21.916196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.916330 (+ 134us) service_pool.cc:227] Handling call
0807 08:01:21.916613 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:22.911166 (+994553us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:23.818105 (+906939us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b54003c3cd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:23.818318 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2179785984}
I20260807 08:01:23.822145 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:23.823249 13709 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:23.331099 16694 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:23.787061 14579 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.019s sys 0.014s
W20260807 08:01:23.835059 14579 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.020s sys 0.014s
I20260807 08:01:23.491334 16167 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:23.620249 16718 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:23.628342 16682 log.cc:924] Time spent T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 1.006s user 0.008s sys 0.013s
W20260807 08:01:23.840126 13769 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.014s sys 0.032s
W20260807 08:01:23.261688 13808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1300) took 1330 ms (client timeout 1526 ms). Trace:
W20260807 08:01:23.748394 14622 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.187s user 0.008s sys 0.038s
I20260807 08:01:23.825419 16698 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:23.855484 16753 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:23.785888 16661 consensus_queue.cc:1035] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
I20260807 08:01:23.862942 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:23.804226 16697 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:23.865945 13702 raft_consensus.cc:1226] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260807 08:01:23.908242 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 157553us
I20260807 08:01:23.896037 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:23.914731 14583 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:23.795295 16686 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:23.802915 14580 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.011s sys 0.006s
I20260807 08:01:23.756393 14618 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:23.750145 16681 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:23.720414 13700 rpcz_store.cc:276] 0807 08:01:21.024648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.172343 (+147695us) service_pool.cc:227] Handling call
0807 08:01:21.172929 (+ 586us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:21.790791 (+617862us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:23.073416 (+1282625us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54000cf154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:23.113240 (+ 39824us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1239195904}
I20260807 08:01:24.032048 13702 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221"
dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:59228
I20260807 08:01:24.032930 13700 raft_consensus.cc:1226] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
W20260807 08:01:24.018824 13585 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.012s sys 0.043s
I20260807 08:01:24.021112 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1638) took 2342 ms. Trace:
W20260807 08:01:24.039233 13585 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s user 0.013s sys 0.043s
W20260807 08:01:23.842542 13808 rpcz_store.cc:269] 0807 08:01:21.931109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.932158 (+ 1049us) service_pool.cc:227] Handling call
0807 08:01:23.164498 (+1232340us) spinlock_profiling.cc:243] Waited 7.4 ms on lock 0x7b5800004298. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00bbce 000000000034aea9 00007f6b21e37a30 00007f6b21efd7b4 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:23.260829 (+ 96331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17010432}
I20260807 08:01:23.920804 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:23.945621 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1312) took 1185 ms. Trace:
I20260807 08:01:24.042160 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:23.935814 13849 debug-util.cc:398] Leaking SignalData structure 0x7b08003a3ee0 after lost signal to thread 13854
I20260807 08:01:24.043804 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.044473 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:24.044934 13805 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:24.022599 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1658) took 1071 ms. Trace:
I20260807 08:01:23.809990 13590 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:24.033408 13702 raft_consensus.cc:595] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:24.045769 13580 rpcz_store.cc:276] 0807 08:01:22.950745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:23.021001 (+ 70256us) service_pool.cc:227] Handling call
0807 08:01:23.021440 (+ 439us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:23.037649 (+ 16209us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:23.037699 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:23.050258 (+ 12559us) log.cc:842] Serialized 26 byte log entry
0807 08:01:23.629030 (+578772us) raft_consensus.cc:1631] Marking committed up to 1
0807 08:01:23.629079 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:23.629153 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:24.021846 (+392693us) raft_consensus.cc:1671] finished
0807 08:01:24.021904 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.022294 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":267420,"threads_started":1,"wal-append.queue_time_us":814678}
I20260807 08:01:23.810520 13706 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:24.035115 13709 raft_consensus.cc:1226] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:23.840579 13769 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.629s user 0.014s sys 0.032s
W20260807 08:01:24.050854 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 839098us
I20260807 08:01:24.042475 13811 rpcz_store.cc:276] 0807 08:01:22.760713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.830725 (+ 70012us) service_pool.cc:227] Handling call
0807 08:01:22.861841 (+ 31116us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:22.862340 (+ 499us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:22.862381 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:22.862926 (+ 545us) log.cc:842] Serialized 26 byte log entry
0807 08:01:23.705013 (+842087us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:23.705065 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:23.705154 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:23.945348 (+240194us) raft_consensus.cc:1671] finished
0807 08:01:23.945371 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:23.945545 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":143,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":4338,"thread_start_us":816377,"threads_started":1,"wal-append.queue_time_us":1049371}
W20260807 08:01:23.813078 13766 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.011s sys 0.022s
I20260807 08:01:23.821611 13701 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:24.054006 13766 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.895s user 0.011s sys 0.022s
W20260807 08:01:24.054394 13849 debug-util.cc:398] Leaking SignalData structure 0x7b08003d1f60 after lost signal to thread 14669
I20260807 08:01:23.926393 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1308) took 1349 ms. Trace:
W20260807 08:01:23.916105 14580 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.628s user 0.011s sys 0.006s
I20260807 08:01:23.833894 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.058035 13807 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
I20260807 08:01:24.058768 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 08:01:24.059027 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1651) took 1084 ms. Trace:
I20260807 08:01:24.059654 13807 rpcz_store.cc:276] 0807 08:01:22.975041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:23.799191 (+824150us) service_pool.cc:227] Handling call
0807 08:01:24.058909 (+259718us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:24.039455 13587 rpcz_store.cc:276] 0807 08:01:21.678832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.678960 (+ 128us) service_pool.cc:227] Handling call
0807 08:01:21.679277 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:21.876890 (+197613us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:21.876925 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:21.877358 (+ 433us) log.cc:842] Serialized 26 byte log entry
0807 08:01:21.996018 (+118660us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:21.996084 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:21.996172 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:23.332543 (+1336371us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:24.019914 (+687371us) raft_consensus.cc:1671] finished
0807 08:01:24.020029 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.020749 (+ 720us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":50833,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":3738,"spinlock_wait_cycles":32829696,"thread_start_us":289495,"threads_started":2,"wal-append.queue_time_us":396127}
I20260807 08:01:24.056414 13817 rpcz_store.cc:276] 0807 08:01:22.577233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.577392 (+ 159us) service_pool.cc:227] Handling call
0807 08:01:22.577792 (+ 400us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:22.954267 (+376475us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:22.954296 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:23.016429 (+ 62133us) log.cc:842] Serialized 26 byte log entry
0807 08:01:23.414496 (+398067us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:23.414552 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:23.414655 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:23.926163 (+511508us) raft_consensus.cc:1671] finished
0807 08:01:23.926185 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:23.926329 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22753,"raft.queue_time_us":73910,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":9050,"thread_start_us":701868,"threads_started":2,"wal-append.queue_time_us":781311}
I20260807 08:01:24.033999 13584 raft_consensus.cc:1261] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:24.066562 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:24.066776 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:24.067996 14625 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:24.068423 14619 raft_consensus.cc:2354] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:24.069850 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:24.070297 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:24.070375 16015 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:23.942214 16678 consensus_meta.cc:287] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.991s user 0.008s sys 0.004s
I20260807 08:01:23.843300 16167 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.104352 14630 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:23.914129 16313 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:23.755244 16359 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:23.195046 16254 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.134336 16762 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:24.075187 16361 consensus_meta.cc:287] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.096s user 0.010s sys 0.005s
I20260807 08:01:24.135460 16361 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.135614 16762 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.136957 16254 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:24.073441 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39124 (request call id 918) took 1153 ms. Trace:
I20260807 08:01:24.138489 16361 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.138693 13706 rpcz_store.cc:276] 0807 08:01:22.919659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.049812 (+1130153us) service_pool.cc:227] Handling call
0807 08:01:24.073078 (+ 23266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:24.140682 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:24.141161 16678 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 08:01:24.142421 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 225270us
I20260807 08:01:23.891958 16396 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.143627 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:24.142989 13706 raft_consensus.cc:1226] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:23.261096 16417 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:23.922966 16429 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.158627 16429 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.160971 16417 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:23.750499 16478 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:23.956930 16502 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:23.933266 16552 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:23.943533 16560 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:23.797235 16596 raft_consensus.cc:1067] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:01:23.728924 16633 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:23.202416 16649 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.198951 16661 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.199261 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:24.199889 16661 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.201035 16649 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:23.948074 16669 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:24.202757 13592 raft_consensus.cc:1226] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:24.204424 16359 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.204600 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1663) took 1169 ms. Trace:
I20260807 08:01:23.207873 16222 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.207765 16746 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.208454 16222 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:23.940755 16679 consensus_meta.cc:287] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.735s user 0.007s sys 0.006s
I20260807 08:01:24.209113 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:24.210521 14620 raft_consensus.cc:2379] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:23.921248 16681 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:24.211098 14620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 460) took 2194 ms (client timeout 1735 ms). Trace:
W20260807 08:01:24.211638 14620 rpcz_store.cc:269] 0807 08:01:22.016315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.017335 (+ 1020us) service_pool.cc:227] Handling call
0807 08:01:24.208775 (+2191440us) spinlock_profiling.cc:243] Waited 9.45 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00bbce 000000000034aea9 00007f6b21e37a30 00007f6b21efd7b4 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:24.211001 (+ 2226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21729152}
I20260807 08:01:24.213045 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1383) took 2123 ms. Trace:
I20260807 08:01:24.213503 13711 rpcz_store.cc:276] 0807 08:01:22.052071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.168039 (+115968us) service_pool.cc:227] Handling call
0807 08:01:22.168418 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:22.612535 (+444117us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.165711 (+1553176us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b5400321c54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:24.175382 (+ 9671us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1007701888}
I20260807 08:01:23.972681 16683 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:23.820310 16688 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:23.945484 16697 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:23.843528 16718 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.224292 16718 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:24.224413 16429 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.227855 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1672) took 1162 ms. Trace:
I20260807 08:01:24.228370 13709 rpcz_store.cc:276] 0807 08:01:23.065239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:23.919648 (+854409us) service_pool.cc:227] Handling call
0807 08:01:23.919871 (+ 223us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:24.049710 (+129839us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.227408 (+177698us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400322154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:24.227748 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":263393152}
I20260807 08:01:23.329463 16750 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:24.229342 16551 consensus_queue.cc:1046] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:24.230587 16551 consensus_queue.cc:1225] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer 7a889001d6494fe5b7cbbe7da1dae302 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260807 08:01:24.230486 16750 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:24.108498 16754 log.cc:924] Time spent T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20260807 08:01:24.134758 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260807 08:01:24.144022 13705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1382) took 2092 ms (client timeout 1517 ms). Trace:
I20260807 08:01:24.104112 16015 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.233903 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:23.331095 16405 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.234108 16015 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.237133 16405 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.135475 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:24.246145 13708 raft_consensus.cc:2396] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 08:01:24.246611 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 264) took 1756 ms. Trace:
I20260807 08:01:24.246434 14618 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:24.248052 16786 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:23.925841 16694 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.152405 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1650) took 1185 ms. Trace:
W20260807 08:01:24.249905 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 106532us
I20260807 08:01:24.140516 16686 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.161350 13812 raft_consensus.cc:1226] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:24.248961 16786 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.249848 13805 rpcz_store.cc:276] 0807 08:01:22.963757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.078846 (+1115089us) service_pool.cc:227] Handling call
0807 08:01:24.106033 (+ 27187us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:24.145228 (+ 39195us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:24.145273 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:24.148314 (+ 3041us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:24.148388 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.148478 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.148623 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:24.251750 13812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1307) took 1693 ms. Trace:
I20260807 08:01:24.252189 13812 rpcz_store.cc:276] 0807 08:01:22.558702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.561865 (+ 3163us) service_pool.cc:227] Handling call
0807 08:01:22.607111 (+ 45246us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:24.250765 (+1643654us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.251468 (+ 703us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54001fccd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:24.251689 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3572824448}
I20260807 08:01:24.252899 16786 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:23.840641 16459 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:24.161785 16698 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.254360 14583 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:24.258150 16552 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:24.258482 13805 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 08:01:24.259464 16544 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:24.259799 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1381) took 2209 ms. Trace:
I20260807 08:01:24.260046 14583 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:24.261577 16393 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:24.262498 13703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1384) took 2209 ms. Trace:
I20260807 08:01:24.263377 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1317) took 1382 ms. Trace:
I20260807 08:01:24.183300 13814 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:24.250257 16762 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.264653 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1646) took 1312 ms. Trace:
I20260807 08:01:24.252213 16694 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.182407 16478 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.190667 16167 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:24.189668 16743 log.cc:924] Time spent T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.246s user 0.000s sys 0.002s
I20260807 08:01:24.260283 13710 rpcz_store.cc:276] 0807 08:01:22.050852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.083822 (+ 32970us) service_pool.cc:227] Handling call
0807 08:01:22.084170 (+ 348us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:22.451141 (+366971us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:22.451184 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:22.475204 (+ 24020us) log.cc:842] Serialized 26 byte log entry
0807 08:01:24.258788 (+1783584us) spinlock_profiling.cc:243] Waited 50.9 ms on lock 0x7b540029c0a0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0af0 00007f6b1cefb028 00007f6b1cf6ba7b 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c
0807 08:01:24.258920 (+ 132us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:24.259045 (+ 125us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.259158 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:24.259254 (+ 96us) raft_consensus.cc:1671] finished
0807 08:01:24.259265 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.259571 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":161795,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":142,"spinlock_wait_cycles":117162368,"thread_start_us":785954,"threads_started":2,"wal-append.queue_time_us":431428,"wal-append.run_cpu_time_us":2081,"wal-append.run_wall_time_us":1027475}
W20260807 08:01:23.921679 16747 log.cc:924] Time spent T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Append to log took a long time: real 0.170s user 0.002s sys 0.000s
I20260807 08:01:24.247054 13708 rpcz_store.cc:276] 0807 08:01:22.490393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.505937 (+ 15544us) service_pool.cc:227] Handling call
0807 08:01:24.135117 (+1629180us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:24.246513 (+111396us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8637184}
I20260807 08:01:24.238842 16661 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.274401 16788 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:24.229187 16550 consensus_queue.cc:1046] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:24.138109 16254 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:24.265280 13808 rpcz_store.cc:276] 0807 08:01:22.951875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.076546 (+1124671us) service_pool.cc:227] Handling call
0807 08:01:24.111948 (+ 35402us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:24.155762 (+ 43814us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:24.162699 (+ 6937us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:24.264193 (+101494us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:24.264320 (+ 127us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.264416 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.264593 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:24.275584 16254 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.278672 16254 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.278813 16788 raft_consensus.cc:686] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:23.748649 16414 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.238832 16750 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:24.281719 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 53786.9us
W20260807 08:01:24.282804 13814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1318) took 1295 ms (client timeout 1589 ms). Trace:
I20260807 08:01:24.280308 16788 consensus_queue.cc:237] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.248646 14618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 641) took 1176 ms. Trace:
W20260807 08:01:24.283257 13814 rpcz_store.cc:269] 0807 08:01:22.987377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:23.726188 (+738811us) service_pool.cc:227] Handling call
0807 08:01:24.282561 (+556373us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b540027abd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:24.282726 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1048148352}
I20260807 08:01:24.283803 14618 rpcz_store.cc:276] 0807 08:01:23.074825 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:23.110144 (+ 35319us) service_pool.cc:227] Handling call
0807 08:01:24.248478 (+1138334us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:24.240006 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1659) took 1284 ms. Trace:
I20260807 08:01:24.288765 13589 rpcz_store.cc:276] 0807 08:01:22.955891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.956050 (+ 159us) service_pool.cc:227] Handling call
0807 08:01:22.960014 (+ 3964us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:22.979412 (+ 19398us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:22.979444 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:22.979976 (+ 532us) log.cc:842] Serialized 26 byte log entry
0807 08:01:23.433767 (+453791us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:24.022581 (+588814us) log.cc:842] Serialized 14 byte log entry
0807 08:01:24.023416 (+ 835us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.023570 (+ 154us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:24.239566 (+215996us) raft_consensus.cc:1671] finished
0807 08:01:24.239593 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.239804 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18924,"prepare.queue_time_us":548119,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":73460,"raft.run_cpu_time_us":439,"raft.run_wall_time_us":439,"thread_start_us":861871,"threads_started":2,"wal-append.queue_time_us":915464}
I20260807 08:01:24.293022 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.295146 16753 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:24.295848 16753 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.240217 13706 raft_consensus.cc:1226] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:24.239943 16679 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:24.232764 13705 rpcz_store.cc:269] 0807 08:01:22.051597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.081262 (+ 29665us) service_pool.cc:227] Handling call
0807 08:01:24.143465 (+2062203us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54003a98d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:24.143722 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3703873408}
I20260807 08:01:24.243000 16633 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:24.243808 13588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1650) took 2471 ms (client timeout 1718 ms). Trace:
I20260807 08:01:24.265959 13804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1302) took 2036 ms. Trace:
W20260807 08:01:24.305436 13588 rpcz_store.cc:269] 0807 08:01:21.772619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.774397 (+ 1778us) service_pool.cc:227] Handling call
0807 08:01:24.217276 (+2442879us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54002053d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:24.243522 (+ 26246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2934820096}
I20260807 08:01:24.244709 16681 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.250810 16686 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.305874 13804 rpcz_store.cc:276] 0807 08:01:22.229616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.238120 (+ 8504us) service_pool.cc:227] Handling call
0807 08:01:23.908039 (+1669919us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b5800004298. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:23.916220 (+ 8181us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:24.165834 (+249614us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:24.165911 (+ 77us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:24.166005 (+ 94us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:24.173351 (+ 7346us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.265479 (+ 92128us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b5400177d54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:24.265553 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.265756 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":525654016}
I20260807 08:01:24.262934 13703 rpcz_store.cc:276] 0807 08:01:22.052529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.204201 (+151672us) service_pool.cc:227] Handling call
0807 08:01:22.204553 (+ 352us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:22.524483 (+319930us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.261905 (+1737422us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54003aacd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:24.262292 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":705219456}
I20260807 08:01:24.307289 16686 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.308593 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:24.310521 14623 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:24.311264 16686 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.253096 13700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1675) took 1185 ms. Trace:
I20260807 08:01:24.194803 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 327) took 1920 ms. Trace:
I20260807 08:01:24.263932 13807 rpcz_store.cc:276] 0807 08:01:22.880907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.060597 (+1179690us) service_pool.cc:227] Handling call
0807 08:01:24.061199 (+ 602us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:24.161537 (+100338us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:24.161577 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:24.161627 (+ 50us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:24.161679 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.263070 (+101391us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b5400115b54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:24.263130 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.263288 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":229901184}
I20260807 08:01:24.298270 16753 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.205232 13592 rpcz_store.cc:276] 0807 08:01:23.034854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:23.052606 (+ 17752us) service_pool.cc:227] Handling call
0807 08:01:23.073176 (+ 20570us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:24.203381 (+1130205us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.204184 (+ 803us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001644d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:24.204512 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2596512512}
I20260807 08:01:24.312768 13816 rpcz_store.cc:276] 0807 08:01:22.274672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.274855 (+ 183us) service_pool.cc:227] Handling call
0807 08:01:22.286753 (+ 11898us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:22.941002 (+654249us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.194173 (+1253171us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b5400114c54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:24.194553 (+ 380us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1443915648}
I20260807 08:01:24.262494 16393 raft_consensus.cc:686] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:24.303591 16679 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:24.304183 16596 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:24.316959 16596 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.316409 16393 consensus_queue.cc:237] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.319219 16596 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.304858 16359 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.320137 16359 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.320587 16429 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:24.321391 16429 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.322755 16359 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:24.324000 16429 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.328966 16787 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:24.329716 16551 raft_consensus.cc:3032] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:24.329639 16787 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.330272 16551 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:24.331801 16787 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.331550 16551 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.332775 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.333458 16551 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:24.354928 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 546310us
I20260807 08:01:24.358034 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.352815 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:24.359220 13587 raft_consensus.cc:2396] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:24.363260 13585 raft_consensus.cc:1226] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260807 08:01:24.363863 13588 raft_consensus.cc:1261] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260807 08:01:24.364078 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 80763.7us
I20260807 08:01:24.364499 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.366318 13592 raft_consensus.cc:1226] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:24.334785 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:24.365453 13537 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:24.357227 13661 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:24.372232 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:24.312624 13700 rpcz_store.cc:276] 0807 08:01:23.067592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.038043 (+970451us) service_pool.cc:227] Handling call
0807 08:01:24.038688 (+ 645us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:24.113713 (+ 75025us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:24.113758 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:24.118232 (+ 4474us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:24.118303 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.252856 (+134553us) spinlock_profiling.cc:243] Waited 71.7 ms on lock 0x7b54003aa2d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:24.252901 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.253037 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":164892544}
I20260807 08:01:24.387503 16769 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.363914 16401 raft_consensus.cc:3032] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:24.388847 16401 raft_consensus.cc:729] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:24.389909 16401 consensus_queue.cc:260] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.391559 16401 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:24.393321 14630 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:24.393944 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1676) took 1326 ms. Trace:
I20260807 08:01:24.396519 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.386868 14622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 464) took 2329 ms. Trace:
I20260807 08:01:24.350800 16750 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:24.368734 16431 consensus_queue.cc:1046] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260807 08:01:24.399082 13769 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:24.401269 16679 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:24.409389 16550 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:24.409858 16550 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:24.410238 16550 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:24.412559 13584 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260807 08:01:24.410238 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 56287.5us
I20260807 08:01:24.414510 16794 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:24.422209 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 67849.8us
I20260807 08:01:24.406469 16686 raft_consensus.cc:2781] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:24.408820 13812 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:24.424510 16254 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:24.450222 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:24.454849 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:24.454705 16417 consensus_queue.cc:1046] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:24.469022 13804 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:24.469357 16417 raft_consensus.cc:3032] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:24.469327 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.469960 16417 raft_consensus.cc:729] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:24.470566 13587 raft_consensus.cc:2379] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:24.471026 16417 consensus_queue.cc:260] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.472409 16417 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:24.475442 16565 consensus_meta.cc:287] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.639s user 0.010s sys 0.006s
I20260807 08:01:24.476375 16459 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:24.478322 13592 raft_consensus.cc:1226] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 3. Ops: []
I20260807 08:01:24.457731 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.481863 16254 raft_consensus.cc:686] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:24.483099 16254 consensus_queue.cc:237] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.487293 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:24.488705 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.489899 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.490356 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.490815 13585 raft_consensus.cc:2379] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:24.491541 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.491663 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.492393 13585 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:24.489897 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.492404 13588 raft_consensus.cc:2396] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 08:01:24.493332 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.415274 16794 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.456713 16429 consensus_queue.cc:1046] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:24.475965 16565 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.416234 16015 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.388177 16769 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:24.495172 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 138071us
I20260807 08:01:24.509490 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.524160 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.462810 16686 raft_consensus.cc:686] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:24.527590 13584 raft_consensus.cc:1226] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260807 08:01:24.507148 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.535614 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:24.532721 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 78101.7us
I20260807 08:01:24.546841 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.549729 13592 raft_consensus.cc:1226] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:24.556351 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:24.559211 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.561401 14627 raft_consensus.cc:2354] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:24.562382 16794 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.562670 16015 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:24.563374 16015 raft_consensus.cc:686] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:24.561283 13807 raft_consensus.cc:1261] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260807 08:01:24.564600 13657 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:24.565325 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 71653.7us
I20260807 08:01:24.565553 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.565999 16565 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.566437 13657 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:24.567538 16512 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:24.567957 16512 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.568145 16661 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:24.568377 16512 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:24.568638 16661 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:24.569073 16661 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:24.546108 16723 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:24.571036 16723 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.571537 16414 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:24.543066 16753 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.031s
I20260807 08:01:24.536477 16459 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:24.574616 16459 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.575163 16753 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:24.575752 16753 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.576931 16459 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.561225 13588 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:24.498037 16750 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.560333 14626 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:24.616106 13816 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 08:01:24.629364 13814 raft_consensus.cc:2396] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260807 08:01:24.632741 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 184451us
I20260807 08:01:24.633385 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:24.638746 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.642088 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.624282 13808 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:24.644465 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.646212 13701 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
W20260807 08:01:24.647034 13701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1679) took 1535 ms (client timeout 1989 ms). Trace:
I20260807 08:01:24.646243 14580 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:24.647538 13701 rpcz_store.cc:269] 0807 08:01:23.111562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:23.254630 (+143068us) service_pool.cc:227] Handling call
0807 08:01:24.646949 (+1392319us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:24.396723 13702 rpcz_store.cc:276] 0807 08:01:23.067944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.048717 (+980773us) service_pool.cc:227] Handling call
0807 08:01:24.049581 (+ 864us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:24.050709 (+ 1128us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:24.050765 (+ 56us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:24.052046 (+ 1281us) log.cc:842] Serialized 52 byte log entry
0807 08:01:24.153272 (+101226us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:24.154381 (+ 1109us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.155399 (+ 1018us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:24.381992 (+226593us) raft_consensus.cc:1671] finished
0807 08:01:24.382016 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.393851 (+ 11835us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":36947,"raft.run_cpu_time_us":634,"raft.run_wall_time_us":47862,"thread_start_us":97544,"threads_started":1,"wal-append.queue_time_us":317980}
I20260807 08:01:24.655144 13804 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:24.657939 13804 raft_consensus.cc:1261] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 08:01:24.658025 16565 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.658831 16649 raft_consensus.cc:3032] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:24.658731 16565 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.659204 16649 raft_consensus.cc:729] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:24.660563 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 462) took 2632 ms (client timeout 1512 ms). Trace:
I20260807 08:01:24.660863 16565 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:24.660975 14626 rpcz_store.cc:269] 0807 08:01:22.028534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.029591 (+ 1057us) service_pool.cc:227] Handling call
0807 08:01:24.198807 (+2169216us) spinlock_profiling.cc:243] Waited 7.39 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:24.660375 (+461568us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54006a04d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:24.660492 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":674129536}
I20260807 08:01:24.564503 16015 consensus_queue.cc:237] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.662465 16762 raft_consensus.cc:3032] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:24.662858 16762 raft_consensus.cc:729] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:24.663815 16762 consensus_queue.cc:260] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.665264 16762 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:24.674965 16550 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.636930 16459 consensus_queue.cc:1046] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260807 08:01:24.627770 16429 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.690372 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.693993 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.638049 16723 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:24.696252 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 57505.9us
I20260807 08:01:24.641816 16753 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.637601 13579 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:24.705299 16429 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.708876 13590 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:24.709899 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:24.710248 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.711678 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.711728 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.712319 13584 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:24.713769 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.714354 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:24.714884 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:24.715780 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.713379 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.719238 13589 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:24.720130 13588 raft_consensus.cc:2379] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:24.722112 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1316) took 1865 ms. Trace:
I20260807 08:01:24.719430 16550 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:24.713260 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:24.695164 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 53010.4us
I20260807 08:01:24.723912 13809 rpcz_store.cc:276] 0807 08:01:22.856606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.989141 (+132535us) service_pool.cc:227] Handling call
0807 08:01:22.989789 (+ 648us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:23.016827 (+ 27038us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:23.016857 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:23.058562 (+ 41705us) log.cc:842] Serialized 26 byte log entry
0807 08:01:24.263875 (+1205313us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b18001b4900. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a025345 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a042e22 00007f6b1cbec5bb 00007f6b1cbf2063 00007f6b1cbf2411 00007f6b1cf3f5cf
0807 08:01:24.625968 (+362093us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:24.626018 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.626089 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:24.721597 (+ 95508us) raft_consensus.cc:1671] finished
0807 08:01:24.721616 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.721887 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":711,"raft.queue_time_us":9224,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":21657,"spinlock_wait_cycles":9171840,"thread_start_us":267152,"threads_started":1,"wal-append.queue_time_us":1626889}
I20260807 08:01:24.753491 16459 consensus_queue.cc:1035] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
W20260807 08:01:24.759159 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 59137.2us
I20260807 08:01:24.725412 16788 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:24.749502 16565 raft_consensus.cc:3032] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:24.769562 16502 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:24.772711 16544 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:24.773543 16544 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.776342 16544 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.779649 16649 consensus_queue.cc:260] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.780943 16649 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:24.788144 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 240612us
I20260807 08:01:24.398826 14622 rpcz_store.cc:276] 0807 08:01:22.056965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.324442 (+267477us) service_pool.cc:227] Handling call
0807 08:01:22.432964 (+108522us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:22.458105 (+ 25141us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:22.468238 (+ 10133us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:22.544210 (+ 75972us) log.cc:842] Serialized 26 byte log entry
0807 08:01:23.866861 (+1322651us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:24.154140 (+287279us) log.cc:842] Serialized 14 byte log entry
0807 08:01:24.386191 (+232051us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.386354 (+ 163us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:24.386422 (+ 68us) raft_consensus.cc:1671] finished
0807 08:01:24.386434 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.386664 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12584,"prepare.queue_time_us":218638,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":3120,"raft.queue_time_us":35414,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":1480037,"threads_started":2,"wal-append.queue_time_us":1278938}
I20260807 08:01:24.792081 16769 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.796264 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1656) took 1864 ms. Trace:
I20260807 08:01:24.796703 13591 rpcz_store.cc:276] 0807 08:01:22.931786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:22.931922 (+ 136us) service_pool.cc:227] Handling call
0807 08:01:22.932251 (+ 329us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:22.932687 (+ 436us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:22.932726 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:22.933370 (+ 644us) log.cc:842] Serialized 52 byte log entry
0807 08:01:24.210709 (+1277339us) spinlock_profiling.cc:243] Waited 7.79 ms on lock 0x7b180029c260. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a025345 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a042e22 00007f6b1cbec5bb 00007f6b1cbf2063 00007f6b1cbf2411 00007f6b1cf3f5cf
0807 08:01:24.563157 (+352448us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:24.563666 (+ 509us) log.cc:842] Serialized 14 byte log entry
0807 08:01:24.563914 (+ 248us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.564044 (+ 130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:24.795806 (+231762us) raft_consensus.cc:1671] finished
0807 08:01:24.795924 (+ 118us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.796088 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":743,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":3933,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":5958,"spinlock_wait_cycles":17923968,"thread_start_us":314640,"threads_started":1,"wal-append.queue_time_us":1763400}
I20260807 08:01:24.798091 16459 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:01:24.801297 16750 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:24.736958 16686 consensus_queue.cc:237] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.802630 16512 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.804754 16512 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.806526 13540 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:24.807749 14580 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:24.808753 16769 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:24.809178 16775 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:24.810907 16694 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:24.811406 16694 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.813700 16429 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:24.814508 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.814610 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.815842 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.817565 16694 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.818645 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.819793 13805 raft_consensus.cc:2396] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:24.761641 16723 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:24.823534 14627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 640) took 1758 ms (client timeout 2208 ms). Trace:
I20260807 08:01:24.825125 14622 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260807 08:01:24.825888 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:24.826869 14627 rpcz_store.cc:269] 0807 08:01:23.065361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:23.065495 (+ 134us) service_pool.cc:227] Handling call
0807 08:01:24.821901 (+1756406us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54005212d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:24.823459 (+ 1558us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3354749184}
W20260807 08:01:24.828212 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 197699us
I20260807 08:01:24.770982 16565 raft_consensus.cc:729] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
W20260807 08:01:24.828187 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 69942.7us
I20260807 08:01:24.831765 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:24.832383 14626 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:24.835944 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:24.773149 16502 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.844326 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.846493 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:24.854770 13661 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:24.851785 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:24.853282 14583 leader_election.cc:334] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.655s (SENT)
I20260807 08:01:24.843971 13808 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 6
I20260807 08:01:24.860306 14583 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:24.853735 16832 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:24.863763 13705 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:24.864303 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 92061.3us
I20260807 08:01:24.848784 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:24.865134 16512 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:24.878175 16666 raft_consensus.cc:2726] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:24.874303 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:24.852954 16723 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.886405 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:24.863744 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:24.891194 13540 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:24.892804 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:24.879284 16775 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:24.876685 16512 raft_consensus.cc:686] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:24.862773 16661 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.897812 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:24.880062 13701 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:24.898414 13807 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:24.896134 13703 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:24.898880 13706 raft_consensus.cc:2379] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:24.883376 16832 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:24.913818 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1645) took 1962 ms. Trace:
I20260807 08:01:24.911978 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 332) took 2038 ms. Trace:
I20260807 08:01:24.916368 13817 rpcz_store.cc:276] 0807 08:01:22.951542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.063083 (+1111541us) service_pool.cc:227] Handling call
0807 08:01:24.063706 (+ 623us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:24.555334 (+491628us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:24.555378 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:24.616125 (+ 60747us) log.cc:842] Serialized 26 byte log entry
0807 08:01:24.741906 (+125781us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:24.741959 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.742052 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:24.913610 (+171558us) raft_consensus.cc:1671] finished
0807 08:01:24.913631 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.913764 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":73274,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":299,"thread_start_us":572036,"threads_started":2,"wal-append.queue_time_us":231898}
I20260807 08:01:24.904618 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:24.919921 13811 rpcz_store.cc:276] 0807 08:01:22.873469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.052734 (+1179265us) service_pool.cc:227] Handling call
0807 08:01:24.077982 (+ 25248us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:24.509866 (+431884us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:24.509910 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:24.526752 (+ 16842us) log.cc:842] Serialized 26 byte log entry
0807 08:01:24.722865 (+196113us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:24.722913 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:24.722998 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:24.911697 (+188699us) raft_consensus.cc:1671] finished
0807 08:01:24.911716 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:24.911887 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21112,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"thread_start_us":545272,"threads_started":2,"wal-append.queue_time_us":317108}
I20260807 08:01:24.926047 14627 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
W20260807 08:01:24.932013 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 78769.6us
I20260807 08:01:24.937892 14583 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:24.901211 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc32413495ee41909c7c5494c8095084"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42248
I20260807 08:01:24.941256 16816 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260807 08:01:24.942905 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 65061.3us
I20260807 08:01:24.939711 14583 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:24.946220 13811 raft_consensus.cc:1226] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:24.948612 16694 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:24.949344 16459 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:24.929546 13702 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:24.893929 16565 consensus_queue.cc:260] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.962201 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:24.955942 14622 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:24.974401 16832 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:24.975368 16565 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:24.934723 16167 consensus_queue.cc:1035] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260807 08:01:24.978104 14583 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:24.965250 16827 log.cc:924] Time spent T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260807 08:01:24.905808 16512 consensus_queue.cc:237] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:24.980391 13540 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:24.982271 14583 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:24.983300 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:24.989670 14626 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 1.
I20260807 08:01:24.991621 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:24.993197 16459 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:24.994516 16832 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:24.990450 16769 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:24.978785 14622 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:24.997723 16502 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:25.020895 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 63038.7us
I20260807 08:01:25.028048 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:25.037928 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:24.997334 14583 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:25.066618 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 69397.8us
I20260807 08:01:25.027143 16723 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:25.084599 13540 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:25.086862 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 184663us
I20260807 08:01:25.039779 16661 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:25.106588 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 50148.7us
W20260807 08:01:25.106192 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 152186us
I20260807 08:01:25.110884 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:25.087958 16393 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:25.039258 16544 consensus_queue.cc:1035] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260807 08:01:25.121882 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:25.128392 14630 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:25.091953 16686 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:25.088439 16723 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:25.159780 13705 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
W20260807 08:01:25.163074 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 52081.3us
I20260807 08:01:25.167048 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:25.168603 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:25.170045 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:25.171378 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:25.167913 13706 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:25.159387 16788 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:25.174760 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:25.178635 14630 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:25.169494 13540 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:25.198928 13580 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:25.206046 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:25.210834 14627 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:25.235320 13703 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:25.177132 13579 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:25.239317 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:25.273404 13708 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
W20260807 08:01:25.277732 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 243557us
I20260807 08:01:25.279088 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:25.171538 16723 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:25.173493 16686 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:25.398290 14626 raft_consensus.cc:1226] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 1. Current term is 2. Ops: []
W20260807 08:01:25.445710 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 237012us
W20260807 08:01:25.401407 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 238560us
I20260807 08:01:25.280468 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:25.357719 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:25.453150 16847 log.cc:924] Time spent T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
I20260807 08:01:25.274402 16816 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:25.269013 16455 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 08:01:25.312505 16801 log.cc:924] Time spent T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
W20260807 08:01:25.433041 16839 log.cc:924] Time spent T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20260807 08:01:25.488590 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:25.491226 13705 raft_consensus.cc:2396] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 08:01:25.491722 13814 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:25.402451 16414 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:25.498080 16843 log.cc:924] Time spent T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260807 08:01:25.498811 16686 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:25.487504 16393 consensus_queue.cc:1046] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260807 08:01:25.269289 16788 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.221s
I20260807 08:01:25.452904 16429 consensus_queue.cc:1046] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260807 08:01:25.513796 13540 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:25.442229 16750 consensus_queue.cc:1046] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260807 08:01:25.487733 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:25.487934 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:25.528752 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:25.496158 16816 raft_consensus.cc:686] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:25.492794 16455 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:25.519738 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 66157.6us
I20260807 08:01:25.532140 13814 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 5
I20260807 08:01:25.537953 14618 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:25.539378 16686 consensus_queue.cc:1046] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:25.540653 16686 raft_consensus.cc:3032] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:25.541047 16686 raft_consensus.cc:729] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:25.541628 16455 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:25.554765 13807 consensus_meta.cc:287] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.651s user 0.010s sys 0.001s
I20260807 08:01:25.555231 13807 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
W20260807 08:01:25.547160 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 53598.6us
I20260807 08:01:25.556875 13537 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:25.527473 16414 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:25.563285 16414 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:25.554646 16694 raft_consensus.cc:3032] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
W20260807 08:01:25.585409 16565 consensus_meta.cc:287] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.610s user 0.012s sys 0.000s
I20260807 08:01:25.577852 16694 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:25.596781 16694 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:25.598129 16694 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:25.599330 16832 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:25.599889 16832 raft_consensus.cc:686] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:25.601451 16836 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:25.603655 16836 raft_consensus.cc:686] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:25.600888 16832 consensus_queue.cc:237] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:25.629757 13540 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:25.606921 13808 consensus_meta.cc:287] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.727s user 0.012s sys 0.000s
I20260807 08:01:25.603945 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:25.647869 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 86837.5us
I20260807 08:01:25.631538 16723 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:25.638832 16661 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:25.639863 13808 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:25.604583 16686 consensus_queue.cc:260] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:25.669809 16401 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:25.630115 16816 consensus_queue.cc:237] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:25.606366 16836 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.100636 16401 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:25.740558 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.101276 16401 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:25.940165 13587 consensus_meta.cc:287] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.855s user 0.010s sys 0.001s
W20260807 08:01:26.102648 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 491533us
W20260807 08:01:25.973254 14622 consensus_meta.cc:287] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.958s user 0.009s sys 0.000s
W20260807 08:01:25.764423 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 185555us
I20260807 08:01:25.741847 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:25.779819 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:25.828521 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 164329us
W20260807 08:01:26.082216 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 466157us
W20260807 08:01:26.085883 13814 consensus_meta.cc:287] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.517s user 0.011s sys 0.000s
I20260807 08:01:26.107908 13814 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:25.676277 16865 log.cc:924] Time spent T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20260807 08:01:26.102691 13587 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:26.104853 13590 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:26.111783 14580 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:26.107107 14622 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:26.069655 16455 consensus_meta.cc:287] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.527s user 0.011s sys 0.000s
W20260807 08:01:25.941475 16877 log.cc:924] Time spent T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
W20260807 08:01:25.790653 16879 log.cc:924] Time spent T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
W20260807 08:01:25.739563 16829 log.cc:924] Time spent T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260807 08:01:26.111996 13657 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:26.011000 16883 raft_consensus.cc:3032] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:25.686295 16863 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:26.038846 16405 raft_consensus.cc:979] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: : Instructing follower 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:25.750916 16679 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.164458 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1665) took 1431 ms. Trace:
I20260807 08:01:26.164775 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:26.165622 13591 raft_consensus.cc:2379] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:26.164842 13816 rpcz_store.cc:276] 0807 08:01:24.732687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.732870 (+ 183us) service_pool.cc:227] Handling call
0807 08:01:24.747338 (+ 14468us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:24.765747 (+ 18409us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:24.765793 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:24.817213 (+ 51420us) log.cc:842] Serialized 26 byte log entry
0807 08:01:25.208635 (+391422us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:25.208676 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:25.208753 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:26.164243 (+955490us) raft_consensus.cc:1671] finished
0807 08:01:26.164279 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:26.164405 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10585,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6911,"thread_start_us":381995,"threads_started":1,"wal-append.queue_time_us":710466}
I20260807 08:01:26.166510 16405 raft_consensus.cc:1067] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Signalling peer 77da774f2bb54cf28302d53d0db224ad to start an election
I20260807 08:01:26.164515 16679 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.170466 16679 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:25.687412 16871 log.cc:924] Time spent T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260807 08:01:25.683772 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:25.682580 16661 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.136626 16850 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:25.779853 16788 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:26.102923 13701 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:26.109627 16686 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:26.153695 14622 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.137344 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:25.999552 16750 raft_consensus.cc:3032] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
W20260807 08:01:26.154093 13418 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 0248ade936924e628c46b8de6252af1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 9210dbc4cc9c45e5b26a463b705d362c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:26.155143 14623 raft_consensus.cc:1226] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:26.142307 16565 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:26.132633 16455 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.157953 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:26.126813 13769 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:26.040602 16753 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:26.183063 16834 consensus_queue.cc:1046] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:26.184110 16649 consensus_queue.cc:1046] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:26.186199 16863 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.125334 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:25.686110 16872 consensus_queue.cc:1046] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.554s
I20260807 08:01:26.018478 16459 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.153255 16883 raft_consensus.cc:729] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:26.201197 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:26.201404 16850 raft_consensus.cc:686] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:26.205412 16788 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:26.208407 13418 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 54544.1us
I20260807 08:01:26.208495 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:26.209630 13817 raft_consensus.cc:3032] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:26.224560 13817 raft_consensus.cc:729] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:26.211107 14622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:42248 (request call id 646) took 1324 ms. Trace:
I20260807 08:01:26.226310 13591 raft_consensus.cc:2379] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:26.226789 14627 raft_consensus.cc:2379] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:26.203207 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1692) took 1068 ms. Trace:
I20260807 08:01:26.221200 13585 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:26.203039 16750 raft_consensus.cc:729] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:26.213060 16753 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.250051 16750 consensus_queue.cc:260] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.251062 16753 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:26.221992 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1644) took 1546 ms. Trace:
I20260807 08:01:26.252050 13587 rpcz_store.cc:276] 0807 08:01:24.672025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.714652 (+ 42627us) service_pool.cc:227] Handling call
0807 08:01:26.218013 (+1503361us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b540021f2d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:26.218109 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":848574208}
I20260807 08:01:26.225348 13817 consensus_queue.cc:260] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.226145 14622 rpcz_store.cc:276] 0807 08:01:24.880741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.886116 (+ 5375us) service_pool.cc:227] Handling call
0807 08:01:26.200430 (+1314314us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":258}
I20260807 08:01:26.221180 16565 raft_consensus.cc:686] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
W20260807 08:01:26.228370 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 115627us
I20260807 08:01:26.234726 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:26.228731 16883 consensus_queue.cc:260] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.229147 16834 raft_consensus.cc:3032] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:26.229493 16788 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.262152 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:26.262722 14627 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:26.229838 16909 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:26.231020 16455 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:26.233675 16863 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.230769 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:26.239190 13811 raft_consensus.cc:3032] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:26.270304 13811 raft_consensus.cc:729] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:26.240005 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.241400 13812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1332) took 1317 ms. Trace:
I20260807 08:01:26.242004 14623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 653) took 1169 ms. Trace:
I20260807 08:01:26.228060 16459 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.251178 16750 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:26.224860 16769 consensus_meta.cc:287] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 1.224s user 0.014s sys 0.000s
I20260807 08:01:26.286600 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:26.287609 14630 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:26.259183 16834 raft_consensus.cc:729] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:26.260713 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:26.295125 14622 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:26.263401 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:26.271781 13817 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:26.296137 13587 raft_consensus.cc:2379] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:26.276546 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.270995 13812 rpcz_store.cc:276] 0807 08:01:24.924159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:25.378740 (+454581us) service_pool.cc:227] Handling call
0807 08:01:25.409220 (+ 30480us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:25.409491 (+ 271us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:25.409537 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:25.480811 (+ 71274us) log.cc:842] Serialized 26 byte log entry
0807 08:01:25.620752 (+139941us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:25.620798 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:25.620881 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:26.241121 (+620240us) raft_consensus.cc:1671] finished
0807 08:01:26.241168 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:26.241330 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":101584,"threads_started":1,"wal-append.queue_time_us":722721}
I20260807 08:01:26.278223 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.258145 13807 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:26.301746 13807 raft_consensus.cc:595] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:26.302191 13807 raft_consensus.cc:603] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:26.307727 13580 raft_consensus.cc:2379] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:26.307646 13769 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:26.306222 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:26.311385 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 203377us
I20260807 08:01:26.313392 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.316136 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:26.317013 14626 raft_consensus.cc:2396] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:26.275345 16909 raft_consensus.cc:686] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:26.290441 16897 raft_consensus.cc:3032] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:26.326601 16661 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.326416 13430 catalog_manager.cc:5495] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:26.328032 16459 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:26.248395 13701 rpcz_store.cc:276] 0807 08:01:25.135224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:25.160892 (+ 25668us) service_pool.cc:227] Handling call
0807 08:01:25.691993 (+531101us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00bbce 000000000034aea9 00007f6b21e37a30 00007f6b21efd7b4 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:26.202466 (+510473us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9581184}
I20260807 08:01:26.329952 13769 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:26.305860 16769 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:26.334369 13769 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:26.334600 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:26.334805 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:26.330514 16902 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:26.342276 16902 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:26.342658 16902 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:26.343401 14627 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:26.338006 14622 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:26.337270 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.346781 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:26.323047 13811 consensus_queue.cc:260] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.332093 13430 catalog_manager.cc:5495] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 3 to 5, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 5 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:26.334821 16892 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:26.355580 13430 catalog_manager.cc:5495] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
W20260807 08:01:26.364229 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 183245us
I20260807 08:01:26.330184 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:26.366120 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:26.330793 16897 raft_consensus.cc:729] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:26.373976 13807 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:26.377965 13807 raft_consensus.cc:595] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Received request to transfer leadership to TS 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:26.375046 13769 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:26.378800 13811 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:26.367910 13430 catalog_manager.cc:5495] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 3 to 4, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:26.338238 16459 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.276180 14623 rpcz_store.cc:276] 0807 08:01:25.072282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:25.084216 (+ 11934us) service_pool.cc:227] Handling call
0807 08:01:25.131267 (+ 47051us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:26.208151 (+1076884us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:26.241741 (+ 33590us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54003309d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:26.241939 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2281385856}
I20260807 08:01:26.381633 13432 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56598 (request call id 300) took 4895 ms. Trace:
I20260807 08:01:26.382072 16897 consensus_queue.cc:260] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.361699 16723 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.367331 16892 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:26.384506 14618 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:26.362057 16850 consensus_queue.cc:237] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.385046 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1672) took 1115 ms. Trace:
I20260807 08:01:26.386127 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:26.385957 16909 consensus_queue.cc:237] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.386755 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:26.388971 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.389863 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.389887 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.294493 16565 consensus_queue.cc:237] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.394423 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:26.397548 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.393972 13430 catalog_manager.cc:5495] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:26.374619 16429 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:26.406279 16905 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:26.407745 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:26.408802 16401 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.413959 13430 catalog_manager.cc:5495] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 4 to 5, leader changed from 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 5 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:26.423983 13430 catalog_manager.cc:5495] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 2 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:26.416671 13592 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:26.410389 13805 rpcz_store.cc:276] 0807 08:01:25.270054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:25.580775 (+310721us) service_pool.cc:227] Handling call
0807 08:01:25.581082 (+ 307us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:26.007407 (+426325us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:26.007451 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:26.008057 (+ 606us) log.cc:842] Serialized 26 byte log entry
0807 08:01:26.318198 (+310141us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:26.318253 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:26.319172 (+ 919us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:26.384831 (+ 65659us) raft_consensus.cc:1671] finished
0807 08:01:26.384855 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:26.384984 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":415866,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":8753,"thread_start_us":598538,"threads_started":2,"wal-append.queue_time_us":271808}
I20260807 08:01:26.396626 16551 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:26.421838 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.422717 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.424585 13807 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b09437796639470caac9548c71d72fb2"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:26.426023 13807 raft_consensus.cc:595] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:26.425978 13589 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:26.426461 13807 raft_consensus.cc:603] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:26.422778 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.427631 14583 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:26.404358 16892 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:26.429641 13430 catalog_manager.cc:5495] T b74446812e814c7e9b4a1bd80bfb485d P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:26.397833 16459 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.422875 16872 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:26.362351 16834 consensus_queue.cc:260] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.417883 16905 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:26.471462 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 145958us
I20260807 08:01:26.418854 16769 consensus_queue.cc:1046] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260807 08:01:26.436785 16429 raft_consensus.cc:686] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:26.428869 16723 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.462451 16897 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:26.496796 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 105422us
I20260807 08:01:26.466044 16872 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:26.479888 16905 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:26.433182 16551 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.500840 13807 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:26.501434 13807 raft_consensus.cc:595] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:26.501770 13807 raft_consensus.cc:603] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:26.395762 13432 rpcz_store.cc:276] 0807 08:01:21.375250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:21.375436 (+ 186us) service_pool.cc:227] Handling call
0807 08:01:26.265117 (+4889681us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:22.000119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:22.216329 (+216210us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:22.223564 (+ 7235us) write_op.cc:435] Acquired schema lock
0807 08:01:22.223596 (+ 32us) tablet.cc:660] Decoding operations
0807 08:01:22.441902 (+218306us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:22.445237 (+ 3335us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:22.450330 (+ 5093us) tablet.cc:683] Acquiring locks for 5 operations
0807 08:01:22.602817 (+152487us) tablet.cc:699] Row locks acquired
0807 08:01:22.604367 (+ 1550us) write_op.cc:260] PREPARE: finished
0807 08:01:22.664447 (+ 60080us) write_op.cc:270] Start()
0807 08:01:22.687349 (+ 22902us) write_op.cc:276] Timestamp: P: 1786089682660933 usec, L: 0
0807 08:01:22.705717 (+ 18368us) op_driver.cc:331] REPLICATION: starting
0807 08:01:22.706940 (+ 1223us) log.cc:842] Serialized 2137 byte log entry
0807 08:01:24.825879 (+2118939us) op_driver.cc:444] REPLICATION: finished
0807 08:01:25.445191 (+619312us) write_op.cc:301] APPLY: starting
0807 08:01:25.445349 (+ 158us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:25.450117 (+ 4768us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:25.450262 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:25.451158 (+ 896us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:25.451496 (+ 338us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0807 08:01:25.452285 (+ 789us) write_op.cc:312] APPLY: finished
0807 08:01:25.452774 (+ 489us) log.cc:842] Serialized 47 byte log entry
0807 08:01:25.454068 (+ 1294us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:25.454348 (+ 280us) write_op.cc:454] Released schema lock
0807 08:01:25.454432 (+ 84us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:25.491184 (+ 36752us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:25.493342 (+ 2158us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":389,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":583124,"apply.run_cpu_time_us":4459,"apply.run_wall_time_us":98659,"num_ops":5,"prepare.queue_time_us":288774,"prepare.run_cpu_time_us":69993,"prepare.run_wall_time_us":1090289,"raft.queue_time_us":486044,"raft.run_cpu_time_us":37396,"raft.run_wall_time_us":323192,"replication_time_us":2116266,"thread_start_us":961437,"threads_started":4,"wal-append.queue_time_us":783858}]]}
I20260807 08:01:26.496659 13579 raft_consensus.cc:2379] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:26.505606 13769 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:26.504868 16904 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:26.506513 16904 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:26.507007 16904 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:26.507609 16919 log.cc:924] Time spent T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20260807 08:01:26.509939 16916 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20260807 08:01:26.513003 16551 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:26.498508 16872 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:26.521026 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:26.519366 16393 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.523492 16884 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:26.523995 16884 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:26.523609 16393 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.524401 16884 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:26.526081 14622 raft_consensus.cc:1226] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:26.515997 16834 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:26.517189 16661 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.536275 13702 raft_consensus.cc:1261] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:26.522238 16883 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:26.543162 13814 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:26.526494 13701 raft_consensus.cc:1226] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:26.549839 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.528229 16775 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.567466 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.559648 13817 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:26.569864 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.570767 13703 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:26.529541 16405 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.573310 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:26.574328 14580 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:26.531435 16902 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.582630 14622 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260807 08:01:26.582491 16459 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:26.584472 16753 raft_consensus.cc:3032] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:26.586210 13706 raft_consensus.cc:1261] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:01:26.550582 16429 consensus_queue.cc:237] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.538056 16769 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.588344 16393 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.590188 13700 raft_consensus.cc:1261] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 08:01:26.590602 13708 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:26.591065 13710 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260807 08:01:26.591162 16775 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.532205 16863 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.606314 13817 raft_consensus.cc:2379] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:26.632804 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:26.641273 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.609567 13705 raft_consensus.cc:1261] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:26.659415 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.685961 13710 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:26.610925 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.677212 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.690112 13708 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:26.695681 13709 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:26.619083 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.699018 13811 raft_consensus.cc:1261] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:26.658681 16902 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:26.703635 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.609009 16459 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:26.700207 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1677) took 1146 ms. Trace:
I20260807 08:01:26.706322 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.706252 13811 rpcz_store.cc:276] 0807 08:01:25.554251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:25.686993 (+132742us) service_pool.cc:227] Handling call
0807 08:01:25.687350 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:26.239100 (+551750us) spinlock_profiling.cc:243] Waited 546 ms on lock 0x7b5400092f20. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cefb730 00007f6b1cf6d300 00007f6b1cf6e371 00007f6b1cf6aedf 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc
0807 08:01:26.699877 (+460777us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:26.700138 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1256063872}
I20260807 08:01:26.657657 13701 raft_consensus.cc:1226] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:26.607146 13769 connection.cc:232] RPC call timeout handler was delayed by -1.5500469789840281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2126s for a response
I20260807 08:01:26.614514 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:26.712426 13769 connection.cc:232] RPC call timeout handler was delayed by -1.4762469788547605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22080000000000002s for a response
I20260807 08:01:26.713701 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.648643 16927 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.714039 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.603932 16401 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.610193 16723 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.620548 16753 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:26.656332 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.614130 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:26.717339 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:26.707226 16880 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260807 08:01:26.605666 16661 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.742059 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.628930 16769 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.723798 13711 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:26.744242 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.744870 13710 raft_consensus.cc:2396] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 08:01:26.745330 13705 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:26.741993 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.745889 13706 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:26.660071 16863 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.601446 16905 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.719449 16940 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.620442 16405 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.768572 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:26.723392 16459 raft_consensus.cc:686] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:26.777827 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.722771 13804 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:26.778469 16459 consensus_queue.cc:237] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.714051 16686 consensus_queue.cc:1035] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260807 08:01:26.764420 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.780001 13702 raft_consensus.cc:2379] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:26.720634 16892 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.756403 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.769861 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.772994 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:26.722373 16927 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.774325 16769 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:26.855651 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:26.854504 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:26.794713 16940 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.783721 16872 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.865139 16872 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:26.867676 16927 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.783938 16902 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:26.869339 13709 raft_consensus.cc:2379] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:26.782069 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.786832 13769 leader_election.cc:398] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:26.871158 13769 leader_election.cc:401] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:26.746747 13700 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:26.780738 16775 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.862098 13705 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:26.844410 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.891477 13711 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:26.823693 13708 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:26.783872 14580 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:26.811298 13706 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:26.865178 16940 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:26.901192 16954 log.cc:924] Time spent T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.147s user 0.000s sys 0.003s
I20260807 08:01:26.903033 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:26.868631 16902 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.784485 13805 raft_consensus.cc:1226] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:26.791095 16904 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.890377 16686 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:26.894691 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:26.896356 16679 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:26.896754 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:26.896957 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.897907 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.909408 13705 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:26.795085 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.898871 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.899595 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.907017 13804 raft_consensus.cc:1226] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:26.893370 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 109573us
I20260807 08:01:26.923548 14580 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:26.924372 13711 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:26.924486 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:26.918110 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 62559us
I20260807 08:01:26.919328 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.926452 13809 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260807 08:01:26.920288 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:26.921473 13710 raft_consensus.cc:2396] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260807 08:01:26.922003 13709 raft_consensus.cc:2379] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:26.796303 16863 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.843022 16661 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:26.916260 16679 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:26.792541 16405 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:26.929916 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.780895 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:26.934028 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:26.892745 14623 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:26.925405 13701 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:26.926841 13809 pending_rounds.cc:76] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 2
I20260807 08:01:26.785713 16905 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:26.938450 14622 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
W20260807 08:01:26.939381 13540 connection.cc:232] RPC call timeout handler was delayed by -3.0453807050362229s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1512s for a response
W20260807 08:01:27.018004 13540 connection.cc:232] RPC call timeout handler was delayed by -3.0408807049971074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
W20260807 08:01:26.942806 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 156131us
W20260807 08:01:26.997236 14583 leader_election.cc:334] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.906s (SENT)
I20260807 08:01:26.995400 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:26.946034 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:26.907047 16892 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:27.007767 13700 raft_consensus.cc:1226] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:27.012516 13702 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:27.021878 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:27.022496 13702 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:26.917443 16884 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:26.857371 16753 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:26.945087 16679 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:26.943943 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:27.026273 14618 raft_consensus.cc:2396] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260807 08:01:27.017640 16897 consensus_meta.cc:287] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.512s user 0.012s sys 0.001s
I20260807 08:01:27.026949 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:26.943431 16666 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:27.027844 13769 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:27.028054 16666 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.029157 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:27.029474 13807 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:27.030133 13769 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:27.030720 16666 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:27.009483 13661 leader_election.cc:398] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
W20260807 08:01:27.031906 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 94628.5us
I20260807 08:01:27.032538 13705 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:26.936563 16872 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
W20260807 08:01:27.018496 13540 connection.cc:232] RPC call timeout handler was delayed by -2.8446807048749179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
I20260807 08:01:27.011843 13809 pending_rounds.cc:97] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260807 08:01:27.035086 13814 raft_consensus.cc:595] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:27.035506 13540 connection.cc:232] RPC call timeout handler was delayed by -2.7024807049892843s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18930000000000002s for a response
I20260807 08:01:27.037985 13814 raft_consensus.cc:603] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:27.036639 13809 raft_consensus.cc:2866] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:27.008638 16902 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:27.013754 13706 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:26.989609 16791 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:27.047597 13812 raft_consensus.cc:1261] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 08:01:27.057677 13661 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.059075 13661 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.060691 13661 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.061455 16723 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:27.062059 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:27.062495 16850 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:27.062985 13816 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:27.065790 14623 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:27.068971 13661 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.054375 16927 consensus_queue.cc:1046] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260807 08:01:27.070698 16405 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:27.043344 16775 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:27.071448 16405 raft_consensus.cc:686] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:27.071671 16775 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.072485 16405 consensus_queue.cc:237] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.070757 16927 raft_consensus.cc:3032] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:27.074945 13661 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.056926 13769 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.087708 13537 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:27.088707 14630 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:27.090341 13769 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:27.047158 13702 raft_consensus.cc:1261] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 08:01:27.095033 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:27.097011 14630 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:27.099634 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.099862 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.100330 13587 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:27.100965 13591 raft_consensus.cc:2396] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260807 08:01:27.065377 16904 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:27.098234 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:27.104460 13585 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:27.099862 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:27.062111 16723 raft_consensus.cc:686] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:27.088140 13808 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:27.110522 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.087574 16850 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:27.112087 14626 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:27.120193 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.121129 13590 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:27.081368 16927 raft_consensus.cc:729] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:27.092276 16872 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:27.110870 13769 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.126600 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:27.126868 14583 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.116904 13592 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:27.129066 14583 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.123772 13579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:56592
I20260807 08:01:27.136687 13579 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:27.137106 13579 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:27.138070 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.141983 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:27.143754 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.088757 16958 raft_consensus.cc:2781] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:01:27.092595 16884 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:27.096124 16905 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:27.146852 14630 raft_consensus.cc:2396] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:27.056155 16902 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260807 08:01:27.105147 16666 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:27.163021 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.105962 16880 raft_consensus.cc:2741] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260807 08:01:27.090026 16897 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260807 08:01:27.179738 13540 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.125145 16872 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:27.103760 16661 consensus_queue.cc:1035] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
W20260807 08:01:27.186421 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 82886.6us
I20260807 08:01:27.191408 13587 raft_consensus.cc:1261] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 08:01:27.159355 16750 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:27.143599 16883 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:27.143404 16940 raft_consensus.cc:2741] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 08:01:27.192449 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:27.207206 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 44286.9us
I20260807 08:01:27.157858 16775 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:27.094094 16791 consensus_queue.cc:1046] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20260807 08:01:27.157058 16958 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:27.164891 16666 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:27.165407 16788 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:27.160430 16905 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:27.174336 16895 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:27.174706 16459 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:27.208412 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.210203 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.174216 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:27.158344 16834 consensus_queue.cc:1046] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260807 08:01:27.218140 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:27.199246 16872 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:27.221443 14618 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:27.223567 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:27.224264 13706 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:27.200924 16694 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:27.202474 16565 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:27.204587 16750 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:27.235384 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:27.228924 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:27.233143 14583 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.237440 16565 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:27.237215 16694 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.237854 16565 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:27.238262 16565 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:27.166712 16950 consensus_queue.cc:1035] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.054s
I20260807 08:01:27.239508 16694 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:27.215731 16927 consensus_queue.cc:260] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:27.247691 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:27.248418 13700 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:27.219718 16788 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.219909 16905 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:27.228765 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:27.251314 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:27.252053 16788 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:27.252417 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:27.254312 16968 log.cc:924] Time spent T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260807 08:01:27.253810 16723 consensus_queue.cc:237] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.258185 14630 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:27.260850 14583 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:27.215246 16666 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:27.281643 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:27.307276 13706 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:27.201843 16551 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
W20260807 08:01:27.309480 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 107894us
I20260807 08:01:27.282759 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:27.314282 13811 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:01:27.310331 13769 leader_election.cc:334] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.536s (SENT)
I20260807 08:01:27.192975 16880 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260807 08:01:27.241355 16753 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:27.287321 16679 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.288766 16791 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:27.290640 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 08:01:27.290275 16872 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.259052 16750 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:27.215135 16958 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:01:27.241227 13415 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet b09437796639470caac9548c71d72fb2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 7a889001d6494fe5b7cbbe7da1dae302 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
W20260807 08:01:27.346647 13415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 105614us
I20260807 08:01:27.307703 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.236655 16775 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 08:01:27.336668 13769 leader_election.cc:334] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.546s (SENT)
I20260807 08:01:27.337257 13587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0248ade936924e628c46b8de6252af1b"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56576
I20260807 08:01:27.337989 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:27.282481 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:27.348768 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:27.340468 14630 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:27.290234 13701 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:27.352865 13661 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:27.363401 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 55795.1us
I20260807 08:01:27.367436 13804 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:27.369685 13579 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:27.334403 16904 raft_consensus.cc:3032] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:27.370159 13805 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 08:01:27.373572 13769 leader_election.cc:334] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.547s (SENT)
I20260807 08:01:27.373982 13587 raft_consensus.cc:595] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Received request to transfer leadership
I20260807 08:01:27.374781 14626 raft_consensus.cc:2379] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:27.341915 16791 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:27.343580 16686 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 08:01:27.387260 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 160393us
I20260807 08:01:27.355525 16850 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:27.393940 13585 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
W20260807 08:01:27.394059 13769 leader_election.cc:334] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.563s (SENT)
I20260807 08:01:27.395581 13769 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:27.390538 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:27.371025 16788 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:27.377825 16904 raft_consensus.cc:729] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:27.426187 13706 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:27.433139 13661 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.422535 16902 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:27.393213 16791 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:27.392249 16863 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 08:01:27.437167 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 72765.1us
I20260807 08:01:27.394824 16401 raft_consensus.cc:2741] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260807 08:01:27.433053 13540 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.435529 16883 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:27.444671 13812 raft_consensus.cc:1226] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:27.379046 16923 consensus_queue.cc:1046] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260807 08:01:27.452243 16979 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260807 08:01:27.457909 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 63877.8us
W20260807 08:01:27.460855 13769 leader_election.cc:334] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.570s (SENT)
I20260807 08:01:27.461294 13769 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.394451 16775 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260807 08:01:27.473801 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:27.397665 16869 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260807 08:01:27.481316 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:27.450759 16863 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:27.483660 13661 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:27.452474 16983 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:27.498168 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:27.503062 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 106102us
I20260807 08:01:27.429000 16850 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.556942 13590 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:27.484108 16979 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:27.569960 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 86383.2us
I20260807 08:01:27.570777 13661 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.569267 16987 raft_consensus.cc:3032] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:27.486748 16895 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:27.480732 16880 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:27.457288 16750 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.489568 16863 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:27.576119 13661 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.562078 13592 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:27.460480 16883 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.572396 16987 raft_consensus.cc:729] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:27.556927 16902 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:27.615276 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 154456us
W20260807 08:01:27.614269 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 106380us
W20260807 08:01:27.622879 13769 leader_election.cc:334] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.701s (SENT)
I20260807 08:01:27.623320 13769 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:27.624886 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 48851.8us
I20260807 08:01:27.575266 16661 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:27.582108 16895 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:27.465993 16904 consensus_queue.cc:260] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.572696 16992 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:27.644097 14583 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.610080 16679 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:27.626055 13811 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
W20260807 08:01:27.652237 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 143880us
I20260807 08:01:27.517102 16950 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
I20260807 08:01:27.676570 16950 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:27.677043 16950 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:27.653882 13661 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:27.594452 16993 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:27.614143 13579 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:27.677742 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:27.677428 16950 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:27.678583 13814 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:27.644296 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:27.680162 13812 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:27.672839 14583 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:27.647107 16723 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:27.693053 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:27.696058 13661 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.652817 16872 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:27.701309 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 85221.5us
W20260807 08:01:27.706209 13769 leader_election.cc:334] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.769s (SENT)
I20260807 08:01:27.668684 16895 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:27.661952 16880 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:27.611690 16565 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.706640 13769 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:27.718506 13769 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.848s (SENT) [suppressed 27 similar messages]
W20260807 08:01:27.719761 13769 leader_election.cc:334] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.879s (SENT)
I20260807 08:01:27.720266 13769 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.721220 13700 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:27.723285 17005 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:27.723997 17005 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.725099 16883 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:27.725859 16883 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 08:01:27.726387 16999 log.cc:924] Time spent T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260807 08:01:27.728149 14618 raft_consensus.cc:1226] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:27.729573 16987 consensus_queue.cc:260] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.731330 16987 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:27.621577 16905 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.742952 16750 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:27.743285 16905 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:27.744164 13809 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:27.700493 16993 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:27.626847 16902 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:27.752691 16902 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:27.703737 16679 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:27.754189 16679 raft_consensus.cc:686] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:27.755366 16679 consensus_queue.cc:237] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.681878 16459 consensus_queue.cc:1046] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260807 08:01:27.763813 16723 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:27.715065 16850 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:27.715826 16694 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:27.670503 16923 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:27.773994 13812 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:27.788321 13812 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:27.789232 13537 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:27.807188 13814 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
W20260807 08:01:27.808398 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 88617us
I20260807 08:01:27.778528 16694 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:27.817284 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 111749us
I20260807 08:01:27.785560 16902 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:27.826300 14580 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:27.815589 16551 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:27.818086 16694 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:27.826929 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:27.838378 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 184193us
W20260807 08:01:27.838732 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 142787us
I20260807 08:01:27.839661 13661 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.841372 13661 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.784837 16923 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:27.864686 13661 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.816746 16990 consensus_queue.cc:1046] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260807 08:01:27.836095 16775 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:27.866868 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.835718 17007 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:27.839768 16565 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:27.838893 16551 raft_consensus.cc:686] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:27.885417 16958 consensus_meta.cc:287] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.511s user 0.011s sys 0.000s
I20260807 08:01:27.885970 16958 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.886394 13540 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:27.887604 16401 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:27.888273 16401 raft_consensus.cc:686] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:27.888409 16958 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:27.889605 16401 consensus_queue.cc:237] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.869876 16981 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:27.894728 13540 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.875861 17007 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.895639 17005 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:27.862914 16405 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:01:27.865813 16723 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:27.896338 16791 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:27.908195 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 63465.4us
W20260807 08:01:27.903863 16753 consensus_meta.cc:287] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.566s user 0.006s sys 0.005s
I20260807 08:01:27.919807 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.909869 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.920593 13591 raft_consensus.cc:2354] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5.
I20260807 08:01:27.900131 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:27.917377 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 47082.9us
W20260807 08:01:27.911070 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 100912us
I20260807 08:01:27.908327 13808 raft_consensus.cc:1226] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:27.923990 16927 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:27.924739 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.925858 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.926100 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:27.926666 13587 raft_consensus.cc:2354] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5.
I20260807 08:01:27.927505 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.928680 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.911281 16923 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:27.882697 16661 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:27.932049 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:27.923919 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:27.922896 13769 leader_election.cc:334] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.939s (SENT)
W20260807 08:01:27.934160 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 64136us
I20260807 08:01:27.929293 13587 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:27.935389 13769 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:27.935832 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:27.936435 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.938091 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.944932 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.903255 17016 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:27.950909 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.929622 16775 raft_consensus.cc:686] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:27.961558 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.962622 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.964073 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.965808 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:27.982270 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:27.982824 13703 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:27.954686 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.951751 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.968129 14626 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:27.983697 14622 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:27.968355 13812 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:01:27.984138 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.908349 16551 consensus_queue.cc:237] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:27.970651 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:27.909893 16816 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260807 08:01:27.914393 16895 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:27.971570 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:27.973551 16694 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.003288 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:28.003968 14623 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:28.004182 16694 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:27.974653 17005 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:28.004899 17005 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.005748 16863 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.007083 17005 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:27.973672 13661 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:27.975586 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.976411 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:28.009215 13814 raft_consensus.cc:2396] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:27.978794 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.979230 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:27.945343 16958 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:27.954341 17026 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:27.969417 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:28.019775 13589 raft_consensus.cc:2396] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260807 08:01:27.988601 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:27.992833 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 69988.5us
I20260807 08:01:27.995790 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.998678 13590 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:28.020618 13587 raft_consensus.cc:2362] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5. Re-sending same reply.
I20260807 08:01:28.021418 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:27.999583 16405 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260807 08:01:28.002127 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:28.012099 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:28.000357 13705 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:28.009303 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:28.013470 13661 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.011714 13706 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:28.014923 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:27.982352 16791 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:28.018339 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:28.022061 13590 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:28.021989 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:28.028450 13587 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:28.021181 13769 leader_election.cc:334] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.982s (SENT)
I20260807 08:01:28.028949 14583 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:28.029103 13661 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.007822 16863 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:27.996217 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:28.026729 13580 raft_consensus.cc:2396] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 08:01:28.030277 13661 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.034430 13591 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 08:01:28.031680 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:28.032406 13588 raft_consensus.cc:1261] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:01:28.035334 13812 raft_consensus.cc:2379] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:28.031114 13702 raft_consensus.cc:3032] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:28.044469 13702 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:28.045241 13702 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.046450 13702 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:01:28.046564 16666 consensus_meta.cc:287] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.713s user 0.012s sys 0.000s
I20260807 08:01:28.046725 16895 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:28.047035 16666 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.048329 16904 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.049103 16666 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:28.051262 13661 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.055171 13587 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:28.056234 13592 raft_consensus.cc:2379] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:28.029913 16661 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:28.063632 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:28.060963 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:28.037143 17026 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:28.051575 16983 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:28.058384 16723 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:28.055642 16979 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:28.055958 16694 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:28.095402 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:28.096410 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:28.115469 13706 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
W20260807 08:01:28.104483 17041 log.cc:924] Time spent T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260807 08:01:28.062312 16863 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:28.084079 16950 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.127435 16950 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:28.127710 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 467) took 1008 ms. Trace:
I20260807 08:01:28.128093 14627 rpcz_store.cc:276] 0807 08:01:27.119680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:27.119881 (+ 201us) service_pool.cc:227] Handling call
0807 08:01:27.164810 (+ 44929us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:27.230588 (+ 65778us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 08:01:27.332576 (+101988us) log.cc:842] Serialized 14 byte log entry
0807 08:01:27.486175 (+153599us) log.cc:842] Serialized 14 byte log entry
0807 08:01:27.583886 (+ 97711us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:27.583959 (+ 73us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:27.584030 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:28.127452 (+543422us) spinlock_profiling.cc:243] Waited 65.5 ms on lock 0x7b5400450454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:28.127507 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:28.127628 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1063,"prepare.queue_time_us":102328,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":5238,"raft.queue_time_us":113565,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":33522,"spinlock_wait_cycles":267224704,"thread_start_us":205430,"threads_started":2,"wal-append.queue_time_us":277607}
I20260807 08:01:28.129317 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:28.130121 13701 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:28.129925 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:28.131389 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:28.131959 13701 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.132133 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:28.105662 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:28.106904 16816 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:28.139899 16816 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.117308 16694 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:28.120111 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:28.084434 17005 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:28.133913 13703 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:28.135157 14630 raft_consensus.cc:2379] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:28.085286 17026 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:28.137722 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:28.144620 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:28.145232 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 124071us
I20260807 08:01:28.146376 16816 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:28.146924 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:28.147089 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:28.114181 17007 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:28.063750 16993 consensus_queue.cc:1035] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.076s
I20260807 08:01:28.147518 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:28.148604 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:28.149663 13584 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:28.036964 16775 consensus_queue.cc:237] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.151929 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:28.121984 16666 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:28.153311 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:28.154527 13706 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:28.162649 16694 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.159766 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:28.160280 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:28.217651 13588 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:28.218044 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 57857us
I20260807 08:01:28.220695 14583 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:28.167172 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:28.221848 13814 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 4
I20260807 08:01:28.183272 13585 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:28.195122 13587 raft_consensus.cc:1261] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 08:01:28.202277 13709 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:28.208175 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:28.156773 17042 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:28.210578 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:28.224011 13805 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:28.231310 13537 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:28.197257 17026 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:28.201364 16723 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:28.208559 13579 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.238663 13661 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.239827 13811 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:28.208768 16750 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:28.211843 17005 raft_consensus.cc:686] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:28.237856 16788 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:28.245548 13805 raft_consensus.cc:1226] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:28.245960 14583 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.229504 13700 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:28.228443 16895 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:28.276499 13814 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:28.279250 13588 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:28.214350 16816 consensus_queue.cc:1046] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260807 08:01:28.224130 16666 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.249878 16834 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:28.276280 16753 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:28.276782 16895 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:28.349828 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 111221us
I20260807 08:01:28.244644 16983 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260807 08:01:28.361066 13661 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:28.241988 17042 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.327322 16686 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:28.252990 16750 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.328760 17025 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:28.364995 17033 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.488s user 0.004s sys 0.001s
I20260807 08:01:28.269968 16905 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260807 08:01:28.250103 16723 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.371546 13661 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:28.369315 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 125171us
I20260807 08:01:28.359978 16895 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.267551 17005 consensus_queue.cc:237] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.356644 16834 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:28.383667 13588 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260807 08:01:28.423166 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 51681.6us
I20260807 08:01:28.426039 13705 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:28.378075 17048 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:28.374143 13702 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:28.383235 16834 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:28.500123 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 77046.5us
W20260807 08:01:28.485100 17033 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260807 08:01:28.377621 17025 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.428833 17067 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:28.489624 16788 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:28.431958 16679 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260807 08:01:28.510795 17067 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:28.539026 17059 log.cc:924] Time spent T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
W20260807 08:01:28.540092 17060 log.cc:924] Time spent T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
W20260807 08:01:28.540977 17054 log.cc:924] Time spent T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.188s user 0.000s sys 0.001s
W20260807 08:01:28.542497 17061 log.cc:924] Time spent T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
W20260807 08:01:28.546089 17058 log.cc:924] Time spent T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.170s user 0.000s sys 0.001s
I20260807 08:01:28.549551 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1365) took 1153 ms. Trace:
I20260807 08:01:28.523089 14618 raft_consensus.cc:1226] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:28.549962 13816 rpcz_store.cc:276] 0807 08:01:27.395861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:27.396449 (+ 588us) service_pool.cc:227] Handling call
0807 08:01:27.426073 (+ 29624us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:27.439932 (+ 13859us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 08:01:27.608343 (+168411us) log.cc:842] Serialized 14 byte log entry
0807 08:01:27.831235 (+222892us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:27.831819 (+ 584us) log.cc:842] Serialized 26 byte log entry
0807 08:01:27.831992 (+ 173us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:27.832021 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:27.832123 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:28.211788 (+379665us) spinlock_profiling.cc:243] Waited 6.84 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:28.549216 (+337428us) raft_consensus.cc:1671] finished
0807 08:01:28.549326 (+ 110us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:28.549464 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7532,"prepare.queue_time_us":370669,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":11299,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":2638,"spinlock_wait_cycles":24072320,"thread_start_us":306728,"threads_started":2,"wal-append.queue_time_us":253992}
I20260807 08:01:28.554071 13700 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:28.558277 16723 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:28.563405 13701 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
W20260807 08:01:28.564244 17024 log.cc:924] Time spent T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.721s user 0.005s sys 0.002s
W20260807 08:01:28.567773 17034 log.cc:924] Time spent T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.691s user 0.000s sys 0.002s
I20260807 08:01:28.569078 13703 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:28.578701 16992 raft_consensus.cc:979] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:01:28.579118 13579 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:28.583067 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1364) took 1202 ms. Trace:
I20260807 08:01:28.583768 13808 rpcz_store.cc:276] 0807 08:01:27.380515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:27.397154 (+ 16639us) service_pool.cc:227] Handling call
0807 08:01:27.460891 (+ 63737us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:27.924225 (+463334us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:28.582733 (+658508us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b5400115b54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:28.582980 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":990324608}
I20260807 08:01:28.584353 14583 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.580830 16661 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:28.586182 16661 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.588246 17025 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:28.589308 17042 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:28.590272 16666 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:28.560855 16694 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.594260 16750 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:28.594198 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:28.594990 13579 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.518687 16788 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.597883 16788 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:28.538028 17067 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.585445 16769 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:28.599354 16694 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:28.599697 16769 raft_consensus.cc:686] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:28.585390 16897 raft_consensus.cc:3032] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:28.602140 16897 raft_consensus.cc:729] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:28.603340 17003 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:28.603624 16990 consensus_queue.cc:1046] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:28.603242 16897 consensus_queue.cc:260] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.604327 17065 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:28.604722 16897 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.605321 16551 raft_consensus.cc:3032] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:28.605803 16551 raft_consensus.cc:729] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:28.605178 16927 consensus_queue.cc:1046] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260807 08:01:28.598373 14622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 476) took 1576 ms (client timeout 1894 ms). Trace:
W20260807 08:01:28.607336 14622 rpcz_store.cc:269] 0807 08:01:27.022436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:27.026648 (+ 4212us) service_pool.cc:227] Handling call
0807 08:01:28.598214 (+1571566us) spinlock_profiling.cc:243] Waited 903 ms on lock 0x7b54002b07d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:28.598316 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2076900096}
I20260807 08:01:28.611155 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:28.630116 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:28.631294 13812 raft_consensus.cc:2379] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:28.635417 13540 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:28.638289 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1696) took 1557 ms. Trace:
I20260807 08:01:28.638696 13809 rpcz_store.cc:276] 0807 08:01:27.080140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:27.080268 (+ 128us) service_pool.cc:227] Handling call
0807 08:01:27.095005 (+ 14737us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:27.744643 (+649638us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:28.637328 (+892685us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b540006bb54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:28.637484 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1492406144}
W20260807 08:01:28.640333 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 208077us
I20260807 08:01:28.643244 14583 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:28.645499 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:28.646806 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:28.647765 13816 raft_consensus.cc:2379] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 2: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:28.661306 14583 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:28.632965 16816 raft_consensus.cc:1067] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Signalling peer 22ff0a1f4a4241d794ab182432c85893 to start an election
I20260807 08:01:28.667919 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:28.668433 16992 raft_consensus.cc:3032] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:28.668887 14630 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:28.669051 16992 raft_consensus.cc:729] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:28.601004 16769 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.671975 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:28.673264 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 52527.7us
I20260807 08:01:28.673385 14583 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.684561 14583 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.606639 16551 consensus_queue.cc:260] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.604871 16990 consensus_queue.cc:1035] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260807 08:01:28.711616 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:28.709733 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:28.661175 16950 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:28.718068 13661 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.709733 13537 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:28.719712 13540 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:28.666898 17015 raft_consensus.cc:2726] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:28.633430 17003 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.677006 17025 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:28.734638 16551 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:01:28.606963 16401 consensus_queue.cc:1159] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7a889001d6494fe5b7cbbe7da1dae302" responder_term: 5 status { last_received { term: 4 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 08:01:28.738142 16834 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.740311 16834 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:28.742969 17065 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:28.670372 16992 consensus_queue.cc:260] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.712425 16979 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:28.765051 13769 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:28.712728 17048 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:28.723080 17016 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:28.725988 17032 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.060s
I20260807 08:01:28.726778 17005 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:28.725548 16816 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:28.725620 16775 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 08:01:28.776613 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 57002.1us
I20260807 08:01:28.786072 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:28.775507 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:28.762712 17015 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:28.774359 16958 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:28.776783 17074 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:28.782687 17005 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:28.809113 13540 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:28.784341 16992 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:28.782282 13579 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:28.800727 16775 raft_consensus.cc:686] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:28.773852 16905 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
W20260807 08:01:28.799431 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 81450.1us
I20260807 08:01:28.777307 17065 consensus_queue.cc:1046] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260807 08:01:28.773459 16979 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.802300 16816 raft_consensus.cc:686] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:28.777048 17077 consensus_queue.cc:1035] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.057s
I20260807 08:01:28.825160 17066 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260807 08:01:28.812736 17005 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.826884 17066 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:28.827260 17066 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:28.827325 13703 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260807 08:01:28.827672 17066 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.802295 16958 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:28.829583 13661 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:28.804438 16769 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:28.779506 17048 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.819792 16990 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:28.844888 13816 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 08:01:28.816881 17064 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:28.824277 17074 raft_consensus.cc:686] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:28.778713 17072 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260807 08:01:28.847532 17062 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 08:01:28.847797 17070 log.cc:924] Time spent T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.180s user 0.001s sys 0.001s
I20260807 08:01:28.848760 13703 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:28.842897 16983 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:28.851250 16893 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:28.852778 16661 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:28.863579 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 63411us
I20260807 08:01:28.907043 14580 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.913313 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:28.865741 17003 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:28.926030 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 150655us
I20260807 08:01:28.838631 16958 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.872579 16775 consensus_queue.cc:237] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.891294 16816 consensus_queue.cc:237] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.891891 16723 consensus_queue.cc:1046] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:28.893257 16979 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:28.807783 17015 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.915212 16983 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:28.930500 13769 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:28.934190 14583 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.856436 16769 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.846612 13540 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:28.940306 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 93764.3us
I20260807 08:01:28.940775 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:28.967027 17097 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:28.974022 13812 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:28.975420 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:28.976226 13812 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:28.984742 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:28.986898 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:28.990819 13809 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 08:01:28.992223 13809 raft_consensus.cc:1226] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:28.994362 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:29.008090 13537 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:29.017747 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:29.018931 13804 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:29.023269 17010 raft_consensus.cc:3032] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:29.023600 13811 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:29.024091 17010 raft_consensus.cc:729] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:29.029731 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 76280.4us
W20260807 08:01:29.031852 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 97762.1us
I20260807 08:01:29.032192 13540 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:29.027117 17083 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:29.033380 13540 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:28.925432 16679 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260807 08:01:28.941411 16769 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:29.037127 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:28.963580 16983 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:28.899816 16895 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:28.941431 17065 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election won for term 3
W20260807 08:01:29.060038 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 52065.6us
I20260807 08:01:29.069626 17074 consensus_queue.cc:237] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:28.929924 16405 consensus_queue.cc:1046] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.113s
I20260807 08:01:29.078332 16459 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:29.078974 17083 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:28.924825 16989 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
I20260807 08:01:28.933759 16893 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.156s
I20260807 08:01:28.962708 16990 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:29.032335 16750 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
W20260807 08:01:29.119180 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 85900.6us
I20260807 08:01:29.080175 16769 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:29.121730 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 89203.5us
I20260807 08:01:29.092236 16666 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:29.033771 16834 consensus_queue.cc:1035] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260807 08:01:29.101622 16923 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 08:01:29.144829 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 213578us
I20260807 08:01:29.086462 17066 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:29.122816 16723 consensus_queue.cc:1225] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer 22ff0a1f4a4241d794ab182432c85893 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260807 08:01:29.147898 16753 consensus_meta.cc:287] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.783s user 0.011s sys 0.004s
I20260807 08:01:29.135998 17097 raft_consensus.cc:686] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:29.146266 16401 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:29.092490 17064 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
W20260807 08:01:29.849077 13537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.033s sys 0.080s
W20260807 08:01:29.855721 13537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.733s user 0.035s sys 0.080s
W20260807 08:01:29.856134 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 747672us
W20260807 08:01:29.839272 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 688649us
I20260807 08:01:29.163872 16666 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:29.474233 17091 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:29.860304 17095 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:29.836478 16850 raft_consensus.cc:3032] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:29.865310 13703 raft_consensus.cc:3032] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:29.897223 13703 raft_consensus.cc:729] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:29.898279 13769 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.934s user 0.007s sys 0.012s
I20260807 08:01:29.865439 16686 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:29.899780 13657 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.969s user 0.008s sys 0.005s
W20260807 08:01:29.900170 13657 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.969s user 0.008s sys 0.005s
W20260807 08:01:29.866991 13540 connection.cc:232] RPC call timeout handler was delayed by -1.7417003600858152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
I20260807 08:01:29.868705 17003 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:29.155210 16990 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:29.393536 16923 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:29.782306 17048 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:29.870576 16723 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:29.876168 13812 raft_consensus.cc:1226] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:29.875888 17097 consensus_queue.cc:237] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:29.865049 16405 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:29.880160 16897 consensus_meta.cc:287] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.275s user 0.005s sys 0.006s
W20260807 08:01:29.936861 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 774171us
W20260807 08:01:29.881923 16992 consensus_meta.cc:287] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.046s user 0.010s sys 0.000s
W20260807 08:01:29.883046 17067 consensus_meta.cc:287] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.284s user 0.014s sys 0.000s
I20260807 08:01:29.885954 17117 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:29.887284 16551 raft_consensus.cc:3032] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Stepping down as leader of term 1
I20260807 08:01:29.188971 16750 raft_consensus.cc:686] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:01:29.898638 13769 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.957s user 0.007s sys 0.012s
W20260807 08:01:29.948640 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 1.00788e+06us
I20260807 08:01:29.912053 16923 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:29.900712 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 972843us
W20260807 08:01:29.475524 17089 log.cc:924] Time spent T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.368s user 0.000s sys 0.000s
W20260807 08:01:29.474480 17080 log.cc:924] Time spent T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.311s user 0.000s sys 0.001s
W20260807 08:01:29.493611 17024 log.cc:924] Time spent T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260807 08:01:29.897989 13703 consensus_queue.cc:260] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:29.797102 17005 consensus_meta.cc:287] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.970s user 0.007s sys 0.000s
W20260807 08:01:29.476472 17109 log.cc:924] Time spent T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260807 08:01:29.903810 13814 raft_consensus.cc:1226] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:29.826285 16927 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:29.838587 16895 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:29.716269 16661 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:29.541610 17065 raft_consensus.cc:686] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:29.896938 16666 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:29.913194 13807 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.924s user 0.006s sys 0.014s
I20260807 08:01:29.912549 13540 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:29.895128 16753 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:29.941126 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:29.900231 16850 raft_consensus.cc:729] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:29.251359 16769 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:29.899905 17062 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:29.946740 14627 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:29.950378 14620 raft_consensus.cc:1226] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:29.908655 17116 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:29.956559 13769 connection.cc:232] RPC call timeout handler was delayed by -2.6482102489098907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2928s for a response
W20260807 08:01:29.383940 17026 consensus_meta.cc:287] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.130s user 0.013s sys 0.000s
I20260807 08:01:29.958169 14625 raft_consensus.cc:1226] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 1. Current term is 2. Ops: []
I20260807 08:01:29.895479 16401 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:29.937070 17014 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:29.963969 13703 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:29.967528 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:29.969364 13817 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:29.970263 14629 raft_consensus.cc:1226] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:29.970921 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:30.121779 14627 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0248ade936924e628c46b8de6252af1b"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
from {username='slave'} at 127.0.0.1:42236
I20260807 08:01:29.946508 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:29.971414 17015 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:29.938602 16405 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:29.973917 17066 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:29.975627 16983 consensus_meta.cc:287] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.862s user 0.003s sys 0.007s
I20260807 08:01:29.941681 16989 consensus_queue.cc:1046] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260807 08:01:29.976159 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:29.946511 17067 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:29.940802 16788 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:29.905663 16686 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:29.948767 16551 raft_consensus.cc:729] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:29.934533 16834 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260807 08:01:29.976471 16750 consensus_queue.cc:237] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.021557 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:30.033221 14623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 474) took 1370 ms. Trace:
I20260807 08:01:29.981892 13587 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 08:01:30.174443 16565 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.021s user 0.008s sys 0.017s
I20260807 08:01:30.087379 13705 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 08:01:30.033999 16927 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:29.963449 17110 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:30.035303 17040 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:30.105625 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 193437us
W20260807 08:01:30.217725 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 283317us
I20260807 08:01:30.223162 13540 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:29.940124 16723 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:29.903419 17095 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:30.110270 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:30.111166 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:29.981657 17005 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:29.951119 17117 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.117201 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:30.096177 17123 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:30.106510 13804 consensus_meta.cc:287] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 1.087s user 0.014s sys 0.010s
I20260807 08:01:30.125566 13702 raft_consensus.cc:1226] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:30.127827 13708 raft_consensus.cc:1226] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260807 08:01:30.131281 13811 raft_consensus.cc:2379] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:30.138599 14627 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:30.108331 14619 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 08:01:30.115420 17016 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.119755 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:30.051847 13807 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.063s user 0.013s sys 0.037s
I20260807 08:01:30.204459 13703 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260807 08:01:30.112725 17062 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.190927 14623 rpcz_store.cc:276] 0807 08:01:28.663320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:28.666435 (+ 3115us) service_pool.cc:227] Handling call
0807 08:01:28.741784 (+ 75349us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:28.742099 (+ 315us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:28.742140 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:28.742765 (+ 625us) log.cc:842] Serialized 26 byte log entry
0807 08:01:28.841937 (+ 99172us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:28.841980 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:28.842066 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:29.978333 (+1136267us) raft_consensus.cc:1671] finished
0807 08:01:30.032823 (+ 54490us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:30.033112 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":90595,"threads_started":1,"wal-append.queue_time_us":175110}
I20260807 08:01:30.190426 17091 raft_consensus.cc:686] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:30.216858 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:30.286814 13705 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:30.287779 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:30.288448 13705 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:30.184296 17015 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.133565 17014 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.229022 14625 raft_consensus.cc:1226] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260807 08:01:30.117463 17026 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:30.203758 16958 consensus_meta.cc:287] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 1.242s user 0.012s sys 0.000s
W20260807 08:01:30.212381 16565 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.055s user 0.008s sys 0.017s
I20260807 08:01:30.192603 17066 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:30.211452 17107 log.cc:924] Time spent T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 1.176s user 0.003s sys 0.013s
I20260807 08:01:30.206959 16401 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:30.207556 16459 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.118111 17116 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:30.203593 17064 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:30.211150 17111 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.272505 14627 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:30.216645 16893 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:30.210554 17127 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:30.273257 13703 pending_rounds.cc:76] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Aborting all ops after (but not including) 4
I20260807 08:01:30.278388 14629 raft_consensus.cc:1226] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260807 08:01:30.366981 13703 pending_rounds.cc:97] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260807 08:01:30.183748 16788 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.155232 16983 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.275193 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:30.314991 17038 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.153106 16850 consensus_queue.cc:260] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.235033 17125 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:30.296628 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:30.270265 17131 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:30.279978 13700 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:30.267778 13804 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:30.320093 14630 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:30.152745 17065 consensus_queue.cc:237] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:30.277707 16990 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:30.324175 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 211489us
I20260807 08:01:30.241880 16987 consensus_queue.cc:1046] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260807 08:01:30.290249 16769 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:30.297250 17134 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.332911 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:30.296803 16661 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:30.268885 17003 consensus_queue.cc:1046] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260807 08:01:30.298588 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:30.235661 17040 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.367338 13703 raft_consensus.cc:2866] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:30.282881 16816 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.052s
I20260807 08:01:30.370901 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:30.290581 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:30.287783 17016 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.237879 17110 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:30.286392 16551 consensus_queue.cc:260] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.280289 17123 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.320613 16897 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.314713 16958 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:30.300027 17015 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.500105 16927 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:30.623025 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:30.754405 17138 log.cc:924] Time spent T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.185s user 0.001s sys 0.002s
I20260807 08:01:30.313309 16753 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.756592 13708 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:30.494382 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1382) took 1646 ms. Trace:
W20260807 08:01:30.419515 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 57911.7us
I20260807 08:01:30.757866 14625 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:30.725946 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:30.477743 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 85453.8us
I20260807 08:01:30.419133 16893 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:31.405427 16989 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:31.406039 16989 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:31.406468 16893 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:30.810432 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:31.407492 13705 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:31.408077 16989 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:31.408443 16816 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:31.408725 13705 raft_consensus.cc:1226] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:31.409372 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1410) took 1272 ms. Trace:
W20260807 08:01:30.822139 13418 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 0248ade936924e628c46b8de6252af1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 77da774f2bb54cf28302d53d0db224ad is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:30.869306 13816 raft_consensus.cc:1226] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:30.468842 13709 raft_consensus.cc:1226] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260807 08:01:30.622507 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:30.488889 14627 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:31.408905 16816 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:30.332252 17066 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:30.623178 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:31.409886 13705 rpcz_store.cc:276] 0807 08:01:30.137192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:31.408210 (+1271018us) service_pool.cc:227] Handling call
0807 08:01:31.408566 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:31.409151 (+ 585us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:31.409322 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:28.980470 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:31.517597 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1718) took 1404 ms. Trace:
I20260807 08:01:31.804399 13709 rpcz_store.cc:276] 0807 08:01:30.113540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.113682 (+ 142us) service_pool.cc:227] Handling call
0807 08:01:30.125864 (+ 12182us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:31.411396 (+1285532us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:31.517168 (+105772us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b54001d33d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:31.517476 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":786809728}
I20260807 08:01:30.364745 16459 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:30.726508 17144 log.cc:924] Time spent T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.243s user 0.000s sys 0.001s
I20260807 08:01:30.451543 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:30.794881 16661 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.866500 17131 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:31.669690 13432 catalog_manager.cc:5495] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 4, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 4 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:31.617019 13430 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57868 (request call id 214) took 6751 ms. Trace:
I20260807 08:01:30.756201 13703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 322) took 1837 ms. Trace:
I20260807 08:01:31.987231 14627 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:30.958449 13661 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.009s sys 0.005s
W20260807 08:01:32.000746 13591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1676) took 1378 ms (client timeout 1560 ms). Trace:
I20260807 08:01:30.978549 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:31.807504 13580 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.052s user 0.000s sys 0.006s
W20260807 08:01:32.151654 13661 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.781s user 0.028s sys 0.028s
I20260807 08:01:32.153187 13814 rpcz_store.cc:276] 0807 08:01:28.848160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:28.851370 (+ 3210us) service_pool.cc:227] Handling call
0807 08:01:28.968464 (+117094us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800004298. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:28.968657 (+ 193us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:29.965455 (+996798us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:30.494068 (+528613us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b54003e3d54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:30.494306 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2173258496}
W20260807 08:01:31.535024 13769 leader_election.cc:334] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 2.176s (SENT)
I20260807 08:01:31.411849 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1412) took 1274 ms. Trace:
I20260807 08:01:32.026960 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 403) took 3116 ms. Trace:
I20260807 08:01:31.127496 13812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1717) took 1161 ms. Trace:
I20260807 08:01:31.307320 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 489) took 1365 ms. Trace:
W20260807 08:01:31.410403 13418 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 588354us
I20260807 08:01:32.150872 13703 rpcz_store.cc:276] 0807 08:01:28.919028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:28.919182 (+ 154us) service_pool.cc:227] Handling call
0807 08:01:28.933976 (+ 14794us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:29.865205 (+931229us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b54000562a0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cefb730 00007f6b1cf6d300 00007f6b1cf6e371 00007f6b1cf6aedf 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc
0807 08:01:30.755945 (+890740us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:30.756144 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2116922240}
I20260807 08:01:32.156745 13584 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
W20260807 08:01:32.151989 13591 rpcz_store.cc:269] 0807 08:01:30.622227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.622725 (+ 498us) service_pool.cc:227] Handling call
0807 08:01:32.000668 (+1377943us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.151453 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:42236 (request call id 490) took 2203 ms. Trace:
I20260807 08:01:32.154173 13661 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:30.821012 17139 log.cc:924] Time spent T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.353s user 0.001s sys 0.000s
I20260807 08:01:30.364682 16834 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:32.336984 14627 rpcz_store.cc:276] 0807 08:01:29.947933 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.106690 (+158757us) service_pool.cc:227] Handling call
0807 08:01:31.856022 (+1749332us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b5400780ad4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf62da7 00007f6b21eaeee0 00007f6b1b5a0784 00007f6b1b5a0712 00007f6b1b5a0641 00007f6b1b5a05bc 00007f6b1b59f872 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:32.151383 (+295361us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37003648}
I20260807 08:01:30.395525 17091 consensus_queue.cc:237] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.620085 17010 consensus_queue.cc:260] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:30.416800 17111 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:30.371095 17064 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:30.868757 16666 consensus_meta.cc:287] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.837s user 0.010s sys 0.004s
I20260807 08:01:30.491519 16405 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.150265 13432 catalog_manager.cc:5495] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } }
W20260807 08:01:32.374883 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 220815us
I20260807 08:01:32.362883 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1392) took 2492 ms. Trace:
W20260807 08:01:31.485349 13585 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.061s user 0.000s sys 0.012s
I20260807 08:01:32.481994 13708 rpcz_store.cc:276] 0807 08:01:30.143025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.272130 (+129105us) service_pool.cc:227] Handling call
0807 08:01:31.411791 (+1139661us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:31.518977 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:32.337620 13812 rpcz_store.cc:276] 0807 08:01:29.966569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:29.966717 (+ 148us) service_pool.cc:227] Handling call
0807 08:01:29.967022 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:30.270320 (+303298us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:30.270350 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:30.270390 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:30.270424 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:31.127225 (+856801us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b54003e5154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:31.127284 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:31.127430 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":697016576}
W20260807 08:01:32.485903 13584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1678) took 1795 ms (client timeout 1848 ms). Trace:
W20260807 08:01:32.153020 13580 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.398s user 0.000s sys 0.009s
I20260807 08:01:30.598356 16863 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.756244 16723 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:31.275709 17124 log.cc:924] Time spent T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 1.174s user 0.001s sys 0.001s
I20260807 08:01:32.336743 14630 rpcz_store.cc:276] 0807 08:01:29.942063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.103744 (+161681us) service_pool.cc:227] Handling call
0807 08:01:31.307075 (+1203331us) spinlock_profiling.cc:243] Waited 81.8 ms on lock 0x7b5400073354. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:31.307241 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":188116352}
I20260807 08:01:31.126719 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 484) took 1205 ms. Trace:
I20260807 08:01:32.528312 14626 rpcz_store.cc:276] 0807 08:01:29.921732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:29.921855 (+ 123us) service_pool.cc:227] Handling call
0807 08:01:29.945198 (+ 23343us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:30.132725 (+187527us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:30.132751 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:30.132798 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:30.132830 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:31.126440 (+993610us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b5400780ad4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:31.126497 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:31.126657 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":411148032}
W20260807 08:01:32.492276 13585 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.073s user 0.000s sys 0.023s
I20260807 08:01:32.478276 13769 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:32.494184 13657 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.618s user 0.007s sys 0.006s
W20260807 08:01:32.531635 13657 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.655s user 0.007s sys 0.006s
I20260807 08:01:31.520668 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1389) took 2653 ms. Trace:
I20260807 08:01:32.532407 13816 rpcz_store.cc:276] 0807 08:01:28.850758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:28.856743 (+ 5985us) service_pool.cc:227] Handling call
0807 08:01:28.857071 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:31.416064 (+2558993us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:31.503938 (+ 87874us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b540037c8d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:31.504197 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4627175168}
I20260807 08:01:32.151841 13811 raft_consensus.cc:2379] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:32.495388 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:32.534754 13590 raft_consensus.cc:2396] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 08:01:32.495954 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:32.495393 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:32.488533 17042 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:31.783666 17040 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:32.550431 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:32.550750 13709 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260807 08:01:32.551100 14630 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:32.552386 13711 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:31.399549 14618 consensus_meta.cc:287] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 1.245s user 0.009s sys 0.005s
I20260807 08:01:32.021415 17159 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:30.492112 17067 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:30.756065 17117 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:30.724685 17062 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:30.503197 17165 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:32.479722 13809 rpcz_store.cc:276] 0807 08:01:29.870630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:29.870789 (+ 159us) service_pool.cc:227] Handling call
0807 08:01:29.871147 (+ 358us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:29.871480 (+ 333us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:29.871508 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:29.871933 (+ 425us) log.cc:842] Serialized 26 byte log entry
0807 08:01:30.160788 (+288855us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:30.160844 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:30.160929 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:32.362563 (+2201634us) raft_consensus.cc:1671] finished
0807 08:01:32.362617 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:32.362807 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":252130,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":37198,"thread_start_us":287524,"threads_started":1,"wal-append.queue_time_us":450721}
W20260807 08:01:30.968645 17142 log.cc:924] Time spent T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.129s user 0.001s sys 0.001s
W20260807 08:01:32.486568 13584 rpcz_store.cc:269] 0807 08:01:30.691042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.691217 (+ 175us) service_pool.cc:227] Handling call
0807 08:01:32.485845 (+1794628us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:31.627367 13808 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.001s sys 0.009s
W20260807 08:01:31.540714 17168 log.cc:924] Time spent T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.272s user 0.000s sys 0.001s
W20260807 08:01:31.019691 16923 consensus_meta.cc:287] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.060s user 0.008s sys 0.007s
W20260807 08:01:31.480517 16987 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.000s sys 0.010s
W20260807 08:01:30.794303 16993 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260807 08:01:32.510417 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:30.824626 17155 log.cc:924] Time spent T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.322s user 0.000s sys 0.002s
W20260807 08:01:31.788089 17163 log.cc:924] Time spent T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20260807 08:01:31.479970 16950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.000s sys 0.005s
I20260807 08:01:32.513049 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1708) took 3620 ms. Trace:
I20260807 08:01:32.622291 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:30.502099 17153 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:31.155835 17146 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:32.495527 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:32.649765 13592 raft_consensus.cc:2379] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:30.752760 17141 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:31.121407 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:31.027154 17177 log.cc:924] Time spent T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.112s user 0.000s sys 0.002s
I20260807 08:01:32.523754 17010 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:31.698288 16791 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:32.524156 17131 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:31.086511 16983 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:31.543455 17110 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:31.219478 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:32.661509 13700 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:31.700947 16679 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.506505 16686 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:31.522122 16893 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:30.866719 13804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1387) took 1961 ms. Trace:
W20260807 08:01:32.525573 14627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 669) took 2287 ms (client timeout 1941 ms). Trace:
I20260807 08:01:32.658161 17016 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:30.506335 17127 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:32.536401 13814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1727) took 2164 ms (client timeout 2080 ms). Trace:
W20260807 08:01:32.681597 13814 rpcz_store.cc:269] 0807 08:01:30.371770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.536140 (+2164370us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:32.536300 (+ 160us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:30.885325 17125 raft_consensus.cc:686] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:31.426712 16788 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:30.452983 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:31.062067 16992 raft_consensus.cc:3032] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:31.110707 17072 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:32.540300 13587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1689) took 1778 ms (client timeout 1776 ms). Trace:
I20260807 08:01:32.541000 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:32.542801 13590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1679) took 1851 ms (client timeout 1873 ms). Trace:
I20260807 08:01:32.695809 13812 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:32.540355 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1723) took 2380 ms. Trace:
W20260807 08:01:32.696509 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1401) took 1213 ms (client timeout 1616 ms). Trace:
W20260807 08:01:32.695588 13587 rpcz_store.cc:269] 0807 08:01:30.761613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:31.481394 (+719781us) service_pool.cc:227] Handling call
0807 08:01:32.540199 (+1058805us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1721728}
W20260807 08:01:32.696897 13812 rpcz_store.cc:269] 0807 08:01:31.482929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.540669 (+1057740us) service_pool.cc:227] Handling call
0807 08:01:32.696454 (+155785us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.698107 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:32.699066 13812 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:01:32.699522 13812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 416) took 2412 ms (client timeout 3124 ms). Trace:
W20260807 08:01:32.699887 13812 rpcz_store.cc:269] 0807 08:01:30.286724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.697798 (+2411074us) service_pool.cc:227] Handling call
0807 08:01:32.699464 (+ 1666us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.700824 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:30.570611 17095 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:30.868904 16927 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:31.173062 16958 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:31.128603 16981 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:30.961014 17116 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:31.529956 16694 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:32.547549 13811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 410) took 2308 ms (client timeout 2227 ms). Trace:
W20260807 08:01:32.704155 13811 rpcz_store.cc:269] 0807 08:01:30.226405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.370567 (+144162us) service_pool.cc:227] Handling call
0807 08:01:32.534043 (+2163476us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.703413 16927 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.705238 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:32.706422 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:31.277240 16976 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:32.707083 16927 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.696732 13805 rpcz_store.cc:276] 0807 08:01:30.160110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.224541 (+ 64431us) service_pool.cc:227] Handling call
0807 08:01:30.273374 (+ 48833us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:32.514766 (+2241392us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:32.514796 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:32.514844 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:32.514876 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:32.540098 (+ 25222us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b5400126e54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:32.540147 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:32.540284 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5154246016}
I20260807 08:01:32.707405 16976 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.708586 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:32.148406 16661 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:30.977959 17014 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:32.709354 13805 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:32.704154 16694 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.710708 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:32.712178 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:32.712736 16694 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:32.548663 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:32.713541 14629 raft_consensus.cc:2396] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 08:01:32.713841 13816 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:32.714443 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1404) took 1152 ms. Trace:
I20260807 08:01:32.714779 13816 rpcz_store.cc:276] 0807 08:01:31.562601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.548305 (+985704us) service_pool.cc:227] Handling call
0807 08:01:32.714382 (+166077us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.715041 16976 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.707125 13811 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:32.715854 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:32.716217 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:32.716866 13811 raft_consensus.cc:595] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:32.549484 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 492) took 1785 ms (client timeout 1601 ms). Trace:
I20260807 08:01:32.717506 13811 raft_consensus.cc:603] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260807 08:01:32.717706 14626 rpcz_store.cc:269] 0807 08:01:30.764092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.549259 (+1785167us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:32.549405 (+ 146us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:32.718811 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:31.859844 17026 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:31.855759 17157 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:32.719947 17157 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.720886 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:32.721452 13811 raft_consensus.cc:595] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:32.721755 13811 raft_consensus.cc:603] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:31.529616 17005 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:32.480610 13807 rpcz_store.cc:276] 0807 08:01:28.910699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:28.988241 (+ 77542us) service_pool.cc:227] Handling call
0807 08:01:28.988592 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:28.988834 (+ 242us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 08:01:30.273342 (+1284508us) log.cc:842] Serialized 14 byte log entry
0807 08:01:32.021130 (+1747788us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:32.021202 (+ 72us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:32.021266 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:32.026675 (+ 5409us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:32.026883 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":833334,"prepare.queue_time_us":319555,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":1197829,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":125,"thread_start_us":1882085,"threads_started":3,"wal-append.queue_time_us":779193}
I20260807 08:01:32.149683 13430 rpcz_store.cc:276] 0807 08:01:24.907200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:24.908801 (+ 1601us) service_pool.cc:227] Handling call
0807 08:01:25.422868 (+514067us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54000b0c54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0af0 00007f6b1cf73f2a 00007f6b22ec5d6c 00007f6b22fbfc3e 00007f6b212d1fb4 00007f6b212d1f42 00007f6b212d1e71 00007f6b212d1dec 00007f6b212d10a2 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:31.616875 (+6194007us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:27.015186 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:27.299860 (+284674us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b5c0001810c. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b232f4501 00007f6b232f43f1 00007f6b232ef12c 00007f6b2098ea59 00007f6b2097a603 00007f6b2097a2b8 00007f6b2097eba1 00007f6b2097eb59 00007f6b2097eae9 00007f6b2097eab1 00007f6b2097ddad
0807 08:01:27.299946 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:27.300004 (+ 58us) write_op.cc:435] Acquired schema lock
0807 08:01:27.300028 (+ 24us) tablet.cc:660] Decoding operations
0807 08:01:27.301220 (+ 1192us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:27.301370 (+ 150us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:27.301412 (+ 42us) tablet.cc:683] Acquiring locks for 8 operations
0807 08:01:27.483424 (+182012us) tablet.cc:699] Row locks acquired
0807 08:01:27.488999 (+ 5575us) write_op.cc:260] PREPARE: finished
0807 08:01:27.576995 (+ 87996us) write_op.cc:270] Start()
0807 08:01:27.763537 (+186542us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b5400002728. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b2348ce91 00007f6b2348bafc 00007f6b2098f304 00007f6b2097a7e3 00007f6b2097a2b8 00007f6b2097eba1 00007f6b2097eb59 00007f6b2097eae9 00007f6b2097eab1 00007f6b2097ddad 00007f6b232a7204
0807 08:01:27.763730 (+ 193us) write_op.cc:276] Timestamp: P: 1786089687566009 usec, L: 0
0807 08:01:27.763752 (+ 22us) op_driver.cc:331] REPLICATION: starting
0807 08:01:27.820735 (+ 56983us) log.cc:842] Serialized 3218 byte log entry
0807 08:01:28.132851 (+312116us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b18002c2b40. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:28.434214 (+301363us) op_driver.cc:444] REPLICATION: finished
0807 08:01:28.773191 (+338977us) write_op.cc:301] APPLY: starting
0807 08:01:28.776258 (+ 3067us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:28.776470 (+ 212us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:28.776485 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:28.869958 (+ 93473us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:28.870167 (+ 209us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0807 08:01:28.870223 (+ 56us) write_op.cc:312] APPLY: finished
0807 08:01:28.870520 (+ 297us) log.cc:842] Serialized 65 byte log entry
0807 08:01:28.870687 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:28.870819 (+ 132us) write_op.cc:454] Released schema lock
0807 08:01:28.870867 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:28.871173 (+ 306us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:28.871202 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":270965,"reported_tablets":19,"spinlock_wait_cycles":397803776,"child_traces":[["op",{"apply.queue_time_us":281622,"apply.run_cpu_time_us":3835,"apply.run_wall_time_us":98355,"num_ops":8,"prepare.queue_time_us":313955,"prepare.run_cpu_time_us":59843,"prepare.run_wall_time_us":951624,"raft.queue_time_us":240559,"raft.run_cpu_time_us":37311,"raft.run_wall_time_us":345255,"replication_time_us":665504,"spinlock_wait_cycles":33952000,"thread_start_us":558603,"threads_started":4,"wal-append.queue_time_us":159471,"wal-append.run_cpu_time_us":26015,"wal-append.run_wall_time_us":1739634}]]}
I20260807 08:01:32.560320 17064 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.724579 13811 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:32.725188 13811 raft_consensus.cc:595] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:32.562287 13659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.131s user 0.005s sys 0.001s
W20260807 08:01:32.725759 13659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.852s user 0.006s sys 0.002s
I20260807 08:01:32.722882 17157 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:30.822893 17112 consensus_queue.cc:1046] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 1.776s
W20260807 08:01:32.351815 14620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 667) took 2120 ms (client timeout 1778 ms). Trace:
I20260807 08:01:32.726388 13811 raft_consensus.cc:603] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260807 08:01:32.726656 14620 rpcz_store.cc:269] 0807 08:01:30.231449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.234388 (+ 2939us) service_pool.cc:227] Handling call
0807 08:01:32.351706 (+2117318us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.727602 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260807 08:01:32.727963 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:32.729045 14620 raft_consensus.cc:2379] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 6: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:32.353788 17204 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:32.729355 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:30.794689 17025 consensus_queue.cc:1046] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 1.754s
I20260807 08:01:32.574144 14583 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:32.509447 16666 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:32.730580 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 156560us
I20260807 08:01:32.731426 16666 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:32.614750 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:32.732118 13809 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:32.540167 16723 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:32.732589 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1407) took 1082 ms. Trace:
W20260807 08:01:32.732965 14583 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 2.227s (SENT) [suppressed 9 similar messages]
I20260807 08:01:32.620738 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:30.827113 17038 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:32.733789 17005 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:32.734431 17005 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:32.735386 17038 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:32.624080 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 1.10287e+06us
I20260807 08:01:32.736990 13706 raft_consensus.cc:2379] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:32.508015 16834 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.738161 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:32.739279 16834 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.557305 14618 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:32.571568 17215 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:32.526875 16863 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:32.621739 13808 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.827s user 0.005s sys 0.012s
W20260807 08:01:32.740075 14618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 480) took 2830 ms (client timeout 2206 ms). Trace:
W20260807 08:01:32.740512 14618 rpcz_store.cc:269] 0807 08:01:29.909498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:29.933695 (+ 24197us) service_pool.cc:227] Handling call
0807 08:01:32.739996 (+2806301us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.741511 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:32.612973 17062 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:32.742224 14618 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:32.742327 17062 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.743646 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:32.744249 14618 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:32.744413 17062 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.351737 17134 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:32.636356 13591 raft_consensus.cc:2396] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260807 08:01:32.745397 13591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1682) took 2052 ms (client timeout 1882 ms). Trace:
W20260807 08:01:32.745750 13591 rpcz_store.cc:269] 0807 08:01:30.692592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.510108 (+1817516us) service_pool.cc:227] Handling call
0807 08:01:32.745324 (+235216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.746068 13705 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 08:01:32.746338 17134 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:32.746675 13591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1685) took 2053 ms (client timeout 1962 ms). Trace:
I20260807 08:01:32.611795 17117 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:32.747213 13588 raft_consensus.cc:2379] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:32.634968 17173 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:32.747483 17117 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.747846 17173 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.749581 17117 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.619705 17208 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:32.750228 17208 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.751749 17173 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.752414 17208 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.479771 13432 catalog_manager.cc:5495] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:32.752633 17005 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:32.752846 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
W20260807 08:01:32.636209 16987 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.762s user 0.000s sys 0.016s
W20260807 08:01:32.643817 16950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.892s user 0.002s sys 0.009s
I20260807 08:01:32.633888 16923 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:32.641461 13580 rpcz_store.cc:276] 0807 08:01:28.892746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:28.893670 (+ 924us) service_pool.cc:227] Handling call
0807 08:01:28.894099 (+ 429us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:28.894476 (+ 377us) raft_consensus.cc:1520] Early marking committed up to index 2
0807 08:01:28.894771 (+ 295us) log.cc:842] Serialized 14 byte log entry
0807 08:01:29.106741 (+211970us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:29.120402 (+ 13661us) log.cc:842] Serialized 26 byte log entry
0807 08:01:29.120559 (+ 157us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:30.210360 (+1089801us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:30.712340 (+501980us) log.cc:842] Serialized 14 byte log entry
0807 08:01:32.488335 (+1775995us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:32.488636 (+ 301us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:32.512839 (+ 24203us) raft_consensus.cc:1671] finished
0807 08:01:32.512858 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:32.512993 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1411431,"prepare.queue_time_us":1353587,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":4707,"raft.queue_time_us":340877,"raft.run_cpu_time_us":464,"raft.run_wall_time_us":76628,"spinlock_wait_cycles":11320064,"thread_start_us":2134335,"threads_started":3,"wal-append.queue_time_us":139815,"wal-append.run_cpu_time_us":68410,"wal-append.run_wall_time_us":3371695}
I20260807 08:01:32.569195 17159 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.612581 17165 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.754652 14620 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:32.755841 17165 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:32.666122 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1398) took 2697 ms. Trace:
I20260807 08:01:32.755362 13432 catalog_manager.cc:5495] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 3 to 4, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 4 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:30.391824 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:32.756168 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:32.679684 13589 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
W20260807 08:01:32.693375 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 127146us
W20260807 08:01:32.759737 13589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1680) took 2067 ms (client timeout 1941 ms). Trace:
W20260807 08:01:32.760089 13589 rpcz_store.cc:269] 0807 08:01:30.691947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.692108 (+ 161us) service_pool.cc:227] Handling call
0807 08:01:32.759670 (+2067562us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27520}
I20260807 08:01:32.703747 16981 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.316500 16459 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:31.125507 17048 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:32.352547 14623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 491) took 2108 ms (client timeout 1656 ms). Trace:
I20260807 08:01:32.565955 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:30.795042 16897 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:32.783569 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 52174.6us
I20260807 08:01:32.615060 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:32.732692 16723 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:32.481545 16753 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.732942 13809 rpcz_store.cc:276] 0807 08:01:31.650456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.574590 (+924134us) service_pool.cc:227] Handling call
0807 08:01:32.732533 (+157943us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.741724 16863 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:32.695729 13701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1712) took 2594 ms (client timeout 1927 ms). Trace:
I20260807 08:01:32.635874 17176 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:32.652798 13592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1681) took 1960 ms (client timeout 1811 ms). Trace:
I20260807 08:01:32.717192 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:32.671343 14627 rpcz_store.cc:269] 0807 08:01:30.238616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.525379 (+2286763us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:32.525488 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:32.757048 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:32.696131 13590 rpcz_store.cc:269] 0807 08:01:30.691594 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.691717 (+ 123us) service_pool.cc:227] Handling call
0807 08:01:32.542715 (+1850998us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12928}
I20260807 08:01:32.757079 13432 catalog_manager.cc:5495] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 3, leader changed from 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:32.758153 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:32.672531 13766 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.820s user 0.019s sys 0.033s
I20260807 08:01:32.758565 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 411) took 2531 ms. Trace:
W20260807 08:01:32.747020 13591 rpcz_store.cc:269] 0807 08:01:30.693487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.746452 (+2052965us) service_pool.cc:214] Skipping call since client already timed out
0807 08:01:32.746566 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260807 08:01:32.656298 17146 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.760954 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:32.762128 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:32.674145 16816 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:32.701498 13812 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:32.767493 14622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 481) took 2786 ms. Trace:
I20260807 08:01:32.756358 13817 rpcz_store.cc:276] 0807 08:01:29.968672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.186390 (+217718us) service_pool.cc:227] Handling call
0807 08:01:30.206137 (+ 19747us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:30.206509 (+ 372us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:30.206548 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:30.207099 (+ 551us) log.cc:842] Serialized 26 byte log entry
0807 08:01:30.527467 (+320368us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:30.527527 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:30.527632 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:32.665832 (+2138200us) raft_consensus.cc:1671] finished
0807 08:01:32.665863 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:32.666045 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11377,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":7692,"thread_start_us":319988,"threads_started":1,"wal-append.queue_time_us":1341360}
I20260807 08:01:32.678536 17066 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:32.673321 17015 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:32.769699 13540 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:32.769934 14580 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:32.695117 17127 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:32.878499 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 108621us
I20260807 08:01:32.788362 17048 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:32.663375 16791 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.879909 17127 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:32.881148 17201 log.cc:924] Time spent T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.188s user 0.000s sys 0.000s
I20260807 08:01:32.695732 16992 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:32.657194 17141 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:32.653703 17153 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:32.668635 16679 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.672287 16893 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.750582 16694 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:32.791730 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1413) took 2648 ms. Trace:
I20260807 08:01:32.874569 16666 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:32.875331 14626 raft_consensus.cc:2379] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:32.875234 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:32.774487 14623 rpcz_store.cc:269] 0807 08:01:30.244267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.332231 (+ 87964us) service_pool.cc:227] Handling call
0807 08:01:32.352419 (+2020188us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:32.826736 13706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1717) took 2719 ms (client timeout 1776 ms). Trace:
I20260807 08:01:32.883541 13816 raft_consensus.cc:2396] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:32.840157 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:32.868633 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 239745us
W20260807 08:01:32.843545 14629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 668) took 2608 ms (client timeout 1927 ms). Trace:
I20260807 08:01:32.884855 13589 raft_consensus.cc:2379] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:32.885529 13805 raft_consensus.cc:2379] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:01:32.885962 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 149697us
I20260807 08:01:32.886107 13432 catalog_manager.cc:5495] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 2, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 2 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260807 08:01:32.887319 13592 rpcz_store.cc:269] 0807 08:01:30.692291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.971017 (+278726us) service_pool.cc:227] Handling call
0807 08:01:32.652711 (+1681694us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1048832}
I20260807 08:01:32.887943 13814 rpcz_store.cc:276] 0807 08:01:30.226804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.757863 (+2531059us) service_pool.cc:227] Handling call
0807 08:01:32.758129 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:32.758289 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:32.758312 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:32.758350 (+ 38us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:32.758380 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:32.758427 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:32.758515 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:32.890295 13766 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.037s user 0.020s sys 0.034s
I20260807 08:01:32.890717 13812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 417) took 1027 ms. Trace:
W20260807 08:01:32.891070 14625 consensus_meta.cc:287] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.552s user 0.008s sys 0.000s
W20260807 08:01:32.891850 13587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1687) took 2131 ms (client timeout 2126 ms). Trace:
I20260807 08:01:32.825909 13584 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:32.824023 14630 raft_consensus.cc:1226] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260807 08:01:32.869953 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 56009.2us
I20260807 08:01:32.891306 14622 rpcz_store.cc:276] 0807 08:01:29.909900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:29.910015 (+ 115us) service_pool.cc:227] Handling call
0807 08:01:29.912217 (+ 2202us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:30.237940 (+325723us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:30.237974 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:30.256137 (+ 18163us) log.cc:842] Serialized 26 byte log entry
0807 08:01:30.711133 (+454996us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:31.410689 (+699556us) log.cc:842] Serialized 14 byte log entry
0807 08:01:31.428190 (+ 17501us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:31.428272 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:32.696430 (+1268158us) raft_consensus.cc:1671] finished
0807 08:01:32.696453 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:32.696571 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70105,"prepare.queue_time_us":42375,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":122495,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":13310,"thread_start_us":861900,"threads_started":3,"wal-append.queue_time_us":473632}
I20260807 08:01:32.671202 13804 rpcz_store.cc:276] 0807 08:01:28.850043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:28.852717 (+ 2674us) service_pool.cc:227] Handling call
0807 08:01:29.017497 (+164780us) spinlock_profiling.cc:243] Waited 4.55 ms on lock 0x7b5800004298. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:30.762862 (+1745365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10469248}
I20260807 08:01:32.767102 17072 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:33.054412 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 284921us
I20260807 08:01:33.175104 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.175587 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:33.175581 13709 rpcz_store.cc:276] 0807 08:01:30.143402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:31.855343 (+1711941us) service_pool.cc:227] Handling call
0807 08:01:31.855834 (+ 491us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:32.551245 (+695411us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:32.791501 (+240256us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54003ab1d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:32.791658 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1597180672}
I20260807 08:01:33.176640 13812 rpcz_store.cc:276] 0807 08:01:31.863794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.700563 (+836769us) service_pool.cc:227] Handling call
0807 08:01:32.890637 (+190074us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.177160 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:33.177878 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.178668 13812 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
W20260807 08:01:33.142194 13711 consensus_meta.cc:287] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.589s user 0.009s sys 0.003s
I20260807 08:01:32.897440 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:33.115599 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1724) took 2938 ms. Trace:
I20260807 08:01:33.055384 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1414) took 2904 ms. Trace:
W20260807 08:01:32.817763 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 202781us
I20260807 08:01:33.181717 13705 rpcz_store.cc:276] 0807 08:01:30.150767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:31.857867 (+1707100us) service_pool.cc:227] Handling call
0807 08:01:31.858233 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:32.746483 (+888250us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.055027 (+308544us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b54001dc4d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:33.055282 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2041118592}
W20260807 08:01:33.178187 13706 rpcz_store.cc:269] 0807 08:01:30.107756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.107914 (+ 158us) service_pool.cc:227] Handling call
0807 08:01:30.978327 (+870413us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:32.826523 (+1848196us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b54000802d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:32.826665 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1368371968}
I20260807 08:01:33.053721 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1707) took 4222 ms. Trace:
I20260807 08:01:33.181388 13807 rpcz_store.cc:276] 0807 08:01:30.176761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.723677 (+2546916us) service_pool.cc:227] Handling call
0807 08:01:32.891040 (+167363us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:32.891211 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:32.891234 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:32.891273 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:32.891308 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:32.891362 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.115353 (+223991us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b540027b0d0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:33.115523 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":384135936}
I20260807 08:01:33.185425 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.187142 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1728) took 1324 ms. Trace:
W20260807 08:01:33.174773 14629 rpcz_store.cc:269] 0807 08:01:30.235187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:31.478267 (+1243080us) service_pool.cc:227] Handling call
0807 08:01:32.828291 (+1350024us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54006815d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:32.843452 (+ 15161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":527256960}
I20260807 08:01:33.187707 13805 rpcz_store.cc:276] 0807 08:01:31.863016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.713255 (+850239us) service_pool.cc:227] Handling call
0807 08:01:33.187080 (+473825us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.189117 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:33.181773 13587 rpcz_store.cc:269] 0807 08:01:30.760583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.757927 (+1997344us) service_pool.cc:227] Handling call
0807 08:01:32.891788 (+133861us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.179873 13711 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
W20260807 08:01:33.190569 13769 connection.cc:232] RPC call timeout handler was delayed by -1.9481752088759094s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1645s for a response
W20260807 08:01:33.191044 13769 connection.cc:232] RPC call timeout handler was delayed by -1.7393752089701593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
W20260807 08:01:33.191501 13769 connection.cc:232] RPC call timeout handler was delayed by -1.6556752088945359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.197s for a response
W20260807 08:01:33.087661 13588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1677) took 2465 ms (client timeout 1778 ms). Trace:
W20260807 08:01:33.192169 13588 rpcz_store.cc:269] 0807 08:01:30.622577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.622718 (+ 141us) service_pool.cc:227] Handling call
0807 08:01:33.087388 (+2464670us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b540036a454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:33.087569 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4581074688}
W20260807 08:01:33.191860 13769 connection.cc:232] RPC call timeout handler was delayed by -1.4045752089004964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
W20260807 08:01:33.193895 13769 connection.cc:232] RPC call timeout handler was delayed by -1.2866414729505777s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
I20260807 08:01:32.877722 16753 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:33.193204 13537 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:33.197924 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:32.789099 17204 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:33.193966 13807 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 1.
W20260807 08:01:33.194209 13769 connection.cc:232] RPC call timeout handler was delayed by -1.2128414728213102s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
I20260807 08:01:33.192603 13432 catalog_manager.cc:5495] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 4, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 4 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:33.200839 13540 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:33.202046 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.183907 13585 rpcz_store.cc:276] 0807 08:01:28.831208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:28.831384 (+ 176us) service_pool.cc:227] Handling call
0807 08:01:28.845452 (+ 14068us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:28.873143 (+ 27691us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:28.873175 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:28.873693 (+ 518us) log.cc:842] Serialized 52 byte log entry
0807 08:01:28.942517 (+ 68824us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:28.965508 (+ 22991us) log.cc:842] Serialized 14 byte log entry
0807 08:01:29.892153 (+926645us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x7b180030e680. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:30.373372 (+481219us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b180030e680. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b19fba4ca 00007f6b2369bf99 00007f6b19e7e990 00007f6b1a040864 00007f6b1a040155 00007f6b20940167 00007f6b1cf75807 00007f6b1cf84f0f 00007f6b1cf84e88
0807 08:01:30.419554 (+ 46182us) log.cc:842] Serialized 14 byte log entry
0807 08:01:32.782713 (+2363159us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.053236 (+270523us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b54003d31d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:33.053295 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:33.053399 (+ 104us) raft_consensus.cc:1671] finished
0807 08:01:33.053411 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.053602 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1730250,"prepare.queue_time_us":1573431,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":116,"raft.queue_time_us":326638,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":10406,"spinlock_wait_cycles":144771456,"thread_start_us":2430662,"threads_started":3,"wal-append.queue_time_us":2239276,"wal-append.run_cpu_time_us":1942,"wal-append.run_wall_time_us":1002306}
I20260807 08:01:33.194408 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:32.896457 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 412) took 2664 ms. Trace:
I20260807 08:01:33.179591 13811 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:33.181341 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.180032 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.178637 14625 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:32.986413 13817 raft_consensus.cc:3032] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:33.196832 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.197145 13591 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:33.200460 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1965102488175035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.45410000000000006s for a response
W20260807 08:01:33.072139 17198 log.cc:924] Time spent T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.446s user 0.000s sys 0.008s
I20260807 08:01:32.828506 14620 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:33.202045 13432 catalog_manager.cc:5495] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:32.813508 17064 consensus_queue.cc:1046] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.406s
I20260807 08:01:32.984941 17125 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:33.056586 16992 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:33.702759 14580 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:33.703749 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.704432 17005 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:33.704762 16992 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:33.705098 17005 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:33.052520 17048 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:33.519492 17223 log.cc:924] Time spent T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260807 08:01:33.701761 17072 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.701990 13817 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:33.513881 17146 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:33.058880 16723 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:32.883812 13701 rpcz_store.cc:269] 0807 08:01:30.086664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.095319 (+ 8655us) service_pool.cc:227] Handling call
0807 08:01:31.121126 (+1025807us) spinlock_profiling.cc:243] Waited 9.95 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:32.680829 (+1559703us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22874880}
I20260807 08:01:33.550516 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:33.517970 14619 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:32.908735 17176 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:33.383308 17157 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:33.380848 17204 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:33.071197 16981 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.378999 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:33.203704 16694 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:33.199419 13705 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:33.756551 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 336) took 1185 ms. Trace:
I20260807 08:01:33.756798 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:33.756888 13705 rpcz_store.cc:276] 0807 08:01:32.571701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.195573 (+623872us) service_pool.cc:227] Handling call
0807 08:01:33.756486 (+560913us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.448948 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.715512 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 414) took 3474 ms. Trace:
I20260807 08:01:33.760589 13805 rpcz_store.cc:276] 0807 08:01:30.240937 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.197849 (+2956912us) service_pool.cc:227] Handling call
0807 08:01:33.714832 (+516983us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.715055 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.715088 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:33.715140 (+ 52us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:33.715241 (+ 101us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.715305 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.715443 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.079385 17003 raft_consensus.cc:3032] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:33.296908 16401 consensus_queue.cc:1046] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.457s
I20260807 08:01:33.207803 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.206110 13588 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:33.766018 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:33.766757 13588 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:01:33.717700 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1363414728548378s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1815s for a response
I20260807 08:01:33.191097 16666 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:33.209932 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.770407 13812 raft_consensus.cc:2379] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:32.882689 16863 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:32.894632 17016 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:33.513440 16923 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:33.279249 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:33.774937 13709 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:33.279253 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.552677 16897 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:33.281400 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.508842 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:33.721290 13537 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:33.284771 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.101766 17159 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.100797 16850 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:33.804023 16893 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:32.832228 13702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1416) took 2626 ms (client timeout 1943 ms). Trace:
W20260807 08:01:33.805058 14619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 671) took 3545 ms (client timeout 1791 ms). Trace:
W20260807 08:01:33.549547 13584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1684) took 2856 ms (client timeout 1931 ms). Trace:
I20260807 08:01:33.224062 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:33.210304 14625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 670) took 2968 ms (client timeout 1655 ms). Trace:
W20260807 08:01:32.974351 17177 log.cc:924] Time spent T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20260807 08:01:33.763701 17157 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:32.828290 16976 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:33.805866 14619 rpcz_store.cc:269] 0807 08:01:30.259926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.362873 (+102947us) service_pool.cc:227] Handling call
0807 08:01:33.804868 (+3441995us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b54006dfe54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:33.804981 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":821035264}
I20260807 08:01:33.806316 16976 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:33.805073 13587 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:33.807447 17157 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.808248 16976 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:33.276877 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.808521 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.809154 13811 raft_consensus.cc:2396] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 08:01:33.810286 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.810513 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 415) took 3529 ms. Trace:
I20260807 08:01:33.810904 13587 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:33.810829 13811 rpcz_store.cc:276] 0807 08:01:30.281328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.809742 (+3528414us) service_pool.cc:227] Handling call
0807 08:01:33.810050 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.810217 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.810241 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:33.810281 (+ 40us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:33.810309 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.810366 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.810462 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.055819 17226 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:32.904462 16459 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:33.817355 13591 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:33.125321 16551 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:01:33.718433 13809 rpcz_store.cc:276] 0807 08:01:30.232192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.895437 (+2663245us) service_pool.cc:227] Handling call
0807 08:01:32.895824 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:32.896037 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:32.896072 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:32.896128 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:32.896169 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:32.896234 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:32.896383 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.819489 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.728219 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:33.722098 14618 raft_consensus.cc:1226] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 08:01:33.701669 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:33.721056 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:33.515043 17153 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:33.731343 13657 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:33.455410 13703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1417) took 3263 ms. Trace:
I20260807 08:01:33.822203 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1716) took 1049 ms. Trace:
W20260807 08:01:33.451910 13711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1713) took 3355 ms (client timeout 1941 ms). Trace:
I20260807 08:01:33.822798 13592 rpcz_store.cc:276] 0807 08:01:32.773359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.448551 (+675192us) service_pool.cc:227] Handling call
0807 08:01:33.822128 (+373577us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:33.822991 13711 rpcz_store.cc:269] 0807 08:01:30.096673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.096841 (+ 168us) service_pool.cc:227] Handling call
0807 08:01:33.451691 (+3354850us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b54002b8c54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:33.451814 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5337144576}
I20260807 08:01:33.824173 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:33.824425 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.824599 14626 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:33.824918 13711 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:33.052814 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 409) took 3197 ms. Trace:
W20260807 08:01:33.822227 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 90973.3us
W20260807 08:01:33.278888 17010 consensus_meta.cc:287] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.618s user 0.012s sys 0.000s
W20260807 08:01:33.278888 13700 consensus_meta.cc:287] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.617s user 0.008s sys 0.004s
W20260807 08:01:33.805250 13702 rpcz_store.cc:269] 0807 08:01:30.190175 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.336509 (+146334us) service_pool.cc:227] Handling call
0807 08:01:32.816458 (+2479949us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.197309 17025 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:33.070891 17123 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.064960 17141 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:33.203526 17218 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:33.750818 13817 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:33.733480 17005 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:33.283567 16927 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:33.433743 16816 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.281s
I20260807 08:01:33.758551 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.841230 13705 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:33.840570 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.841928 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1426) took 1071 ms. Trace:
I20260807 08:01:33.842396 13705 rpcz_store.cc:276] 0807 08:01:32.770896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.758270 (+987374us) service_pool.cc:227] Handling call
0807 08:01:33.841850 (+ 83580us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.843520 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.460487 17190 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:33.219365 16958 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:33.845387 13817 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:33.845901 13592 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:33.282312 17216 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:33.408723 16791 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:33.737416 14620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 478) took 1247 ms. Trace:
I20260807 08:01:33.218152 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:33.746626 17074 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:33.769304 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.779412 17016 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:33.460933 17127 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.299s
I20260807 08:01:33.203783 17179 consensus_queue.cc:1046] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.560s
I20260807 08:01:33.767650 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.774654 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.383270 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.775846 13585 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:33.776800 14622 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:33.773356 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.778599 16694 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:33.836338 13700 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:33.780961 13769 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:33.855201 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 325) took 1321 ms. Trace:
W20260807 08:01:33.860805 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 79956.9us
I20260807 08:01:33.861177 13705 rpcz_store.cc:276] 0807 08:01:32.525418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.843223 (+1317805us) service_pool.cc:227] Handling call
0807 08:01:33.846824 (+ 3601us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.781585 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 330) took 1241 ms. Trace:
I20260807 08:01:33.862423 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:33.862619 13706 rpcz_store.cc:276] 0807 08:01:32.539719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.195570 (+655851us) service_pool.cc:227] Handling call
0807 08:01:33.781501 (+585931us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.862435 13769 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:33.863796 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:33.786926 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 65742.7us
I20260807 08:01:33.769412 16679 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:33.796249 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:33.770707 17116 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:33.844915 16927 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:33.772132 16863 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:33.798470 17176 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.799830 17204 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.800285 16990 raft_consensus.cc:3032] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:33.801396 16666 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:33.812153 13807 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:33.887972 13807 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:33.888854 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1403) took 2370 ms. Trace:
I20260807 08:01:33.812093 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:33.812424 13584 rpcz_store.cc:269] 0807 08:01:30.693193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.620289 (+1927096us) service_pool.cc:227] Handling call
0807 08:01:33.549254 (+928965us) spinlock_profiling.cc:243] Waited 92.2 ms on lock 0x7b54001e3a54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:33.549429 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":212047744}
I20260807 08:01:33.780817 14627 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:33.764933 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:33.815128 14625 rpcz_store.cc:269] 0807 08:01:30.242336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.298336 (+ 56000us) service_pool.cc:227] Handling call
0807 08:01:33.210212 (+2911876us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.821497 14630 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:32.840484 17026 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:33.830216 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1725) took 3539 ms. Trace:
I20260807 08:01:33.822386 13703 rpcz_store.cc:276] 0807 08:01:30.191802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.509196 (+2317394us) service_pool.cc:227] Handling call
0807 08:01:32.509644 (+ 448us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:32.525691 (+ 16047us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:32.525719 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:32.526209 (+ 490us) log.cc:842] Serialized 26 byte log entry
0807 08:01:32.845930 (+319721us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:33.355674 (+509744us) log.cc:842] Serialized 14 byte log entry
0807 08:01:33.455021 (+ 99347us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.455122 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:33.455177 (+ 55us) raft_consensus.cc:1671] finished
0807 08:01:33.455194 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.455342 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":357510,"prepare.queue_time_us":872538,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":50,"raft.queue_time_us":160832,"raft.run_cpu_time_us":4742,"raft.run_wall_time_us":82286,"thread_start_us":641288,"threads_started":2,"wal-append.queue_time_us":499392}
I20260807 08:01:33.825309 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 331) took 1285 ms. Trace:
I20260807 08:01:33.892328 13811 rpcz_store.cc:276] 0807 08:01:30.290587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.811749 (+3521162us) service_pool.cc:227] Handling call
0807 08:01:33.829568 (+ 17819us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.829817 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.829856 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:33.829906 (+ 50us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:33.829953 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.830018 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.830148 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.831341 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.871678 17218 raft_consensus.cc:3032] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:33.833423 13657 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:33.837946 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.895530 13589 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:33.813323 16897 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:33.817965 16459 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:33.818248 17226 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:33.837774 17025 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:33.902102 17025 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:33.848109 14629 raft_consensus.cc:2396] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:33.886359 16990 raft_consensus.cc:729] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:33.851926 17216 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:33.903515 16990 consensus_queue.cc:260] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:33.905088 17216 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.899523 16459 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:33.852394 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.906395 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 337) took 1334 ms. Trace:
I20260807 08:01:33.906769 13702 rpcz_store.cc:276] 0807 08:01:32.572063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.844930 (+1272867us) service_pool.cc:227] Handling call
0807 08:01:33.906319 (+ 61389us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:33.779719 13709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 338) took 1207 ms (client timeout 1521 ms). Trace:
W20260807 08:01:33.908620 13709 rpcz_store.cc:269] 0807 08:01:32.572391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.191218 (+618827us) service_pool.cc:227] Handling call
0807 08:01:33.779655 (+588437us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.909807 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.910079 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1405) took 2325 ms. Trace:
I20260807 08:01:33.910540 13709 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:33.910642 13811 rpcz_store.cc:276] 0807 08:01:31.584724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.893328 (+2308604us) service_pool.cc:227] Handling call
0807 08:01:33.909417 (+ 16089us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.909667 (+ 250us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.909706 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:33.909763 (+ 57us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:33.909813 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.909881 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.910011 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.911896 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.854426 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.912981 13588 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:33.913074 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1406) took 2290 ms. Trace:
I20260807 08:01:33.838358 17166 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:33.855926 13814 raft_consensus.cc:2396] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 08:01:33.856364 13805 raft_consensus.cc:1226] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 08:01:33.858336 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33050 (request call id 436) took 1492 ms. Trace:
I20260807 08:01:33.920104 13585 rpcz_store.cc:276] 0807 08:01:32.364809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.278969 (+914160us) service_pool.cc:227] Handling call
0807 08:01:33.857465 (+578496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.920295 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.858022 17065 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260807 08:01:33.869047 17157 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:33.828429 13808 rpcz_store.cc:276] 0807 08:01:29.855235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:29.856911 (+ 1676us) service_pool.cc:227] Handling call
0807 08:01:29.886664 (+ 29753us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:30.325473 (+438809us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:30.335922 (+ 10449us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:30.534600 (+198678us) log.cc:842] Serialized 26 byte log entry
0807 08:01:32.740586 (+2205986us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:32.740628 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:32.740720 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:33.052449 (+311729us) raft_consensus.cc:1671] finished
0807 08:01:33.052476 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.052727 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":298161,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":14647,"thread_start_us":2400991,"threads_started":2,"wal-append.queue_time_us":2022848}
I20260807 08:01:33.846997 17190 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:33.931558 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1400) took 3609 ms. Trace:
I20260807 08:01:33.932073 13816 rpcz_store.cc:276] 0807 08:01:30.321814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.850582 (+3528768us) service_pool.cc:227] Handling call
0807 08:01:33.850960 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.905900 (+ 54940us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.905931 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:33.905974 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:33.906006 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.931265 (+ 25259us) spinlock_profiling.cc:243] Waited 54.7 ms on lock 0x7b540027c4d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:33.931327 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.931487 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":125856128}
I20260807 08:01:33.869750 17066 consensus_queue.cc:1046] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:33.763456 17003 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:33.865100 13537 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:33.871701 13705 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:33.939189 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 418) took 2075 ms. Trace:
I20260807 08:01:33.939505 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 327) took 1400 ms. Trace:
I20260807 08:01:33.939953 13705 rpcz_store.cc:276] 0807 08:01:32.538755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.862114 (+1323359us) service_pool.cc:227] Handling call
0807 08:01:33.939429 (+ 77315us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.941076 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.879479 17123 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:33.879845 17159 consensus_queue.cc:1046] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:33.880580 17215 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:33.849594 14620 rpcz_store.cc:276] 0807 08:01:32.490103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.755660 (+265557us) service_pool.cc:227] Handling call
0807 08:01:33.733775 (+978115us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:33.871021 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 74850.8us
I20260807 08:01:33.948028 13661 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:33.949323 13661 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:33.847509 14618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 493) took 3047 ms. Trace:
I20260807 08:01:33.848227 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.889343 13807 rpcz_store.cc:276] 0807 08:01:31.518221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.887163 (+2368942us) service_pool.cc:227] Handling call
0807 08:01:33.887564 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.888558 (+ 994us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.888791 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.892939 13711 rpcz_store.cc:276] 0807 08:01:32.568197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.823939 (+1255742us) service_pool.cc:227] Handling call
0807 08:01:33.825258 (+ 1319us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:32.841040 17111 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.883558 16863 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:33.882745 16927 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:33.850780 16958 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:33.897955 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 64669us
I20260807 08:01:33.897549 16897 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:33.901559 17226 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:33.901988 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.832755 16976 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:33.855569 13812 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 2.
I20260807 08:01:33.907984 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.913856 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.914489 17204 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 08:01:33.883869 13700 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1418) took 3674 ms (client timeout 4434 ms). Trace:
I20260807 08:01:33.914507 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.916592 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.917490 13817 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260807 08:01:33.970438 16976 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:33.922334 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:33.905156 16990 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:33.924160 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1402) took 2419 ms. Trace:
I20260807 08:01:33.999022 13805 rpcz_store.cc:276] 0807 08:01:31.505172 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.855717 (+2350545us) service_pool.cc:227] Handling call
0807 08:01:33.856040 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:33.923820 (+ 67780us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.924089 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.000905 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1730) took 1378 ms. Trace:
I20260807 08:01:33.921293 13703 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:33.927424 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1408) took 2228 ms. Trace:
I20260807 08:01:33.935477 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:34.002911 13585 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 2.
I20260807 08:01:33.894840 17218 raft_consensus.cc:729] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:34.002700 13814 rpcz_store.cc:276] 0807 08:01:31.699057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.926465 (+2227408us) service_pool.cc:227] Handling call
0807 08:01:33.926832 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.927042 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.927077 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:33.927133 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:33.927176 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.927241 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.927360 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.878602 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:34.004107 13706 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:34.004606 13814 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 08:01:33.942354 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:33.943188 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 78193.3us
I20260807 08:01:33.946031 13709 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:33.856325 17074 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:33.966588 16897 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:33.928314 17166 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:33.946637 16816 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 08:01:33.964627 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 246898us
I20260807 08:01:34.013622 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:34.013871 13540 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:33.960501 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 421) took 1453 ms. Trace:
W20260807 08:01:33.977216 13700 rpcz_store.cc:269] 0807 08:01:30.207972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.952841 (+744869us) service_pool.cc:227] Handling call
0807 08:01:33.882062 (+2929221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.014493 13807 rpcz_store.cc:276] 0807 08:01:32.506738 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.959537 (+1452799us) service_pool.cc:227] Handling call
0807 08:01:33.959919 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.960120 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.960155 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:33.960212 (+ 57us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:33.960253 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.960315 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.960438 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.015359 17016 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.015868 17016 raft_consensus.cc:2726] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:34.016273 13700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1423) took 3650 ms. Trace:
I20260807 08:01:34.016682 13700 rpcz_store.cc:276] 0807 08:01:30.365633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.015482 (+3649849us) service_pool.cc:227] Handling call
0807 08:01:34.015774 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.015939 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.015969 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.016011 (+ 42us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:34.016043 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.016096 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.016219 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.938071 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1427) took 1143 ms. Trace:
I20260807 08:01:34.017815 13701 rpcz_store.cc:276] 0807 08:01:32.791143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.758139 (+966996us) service_pool.cc:227] Handling call
0807 08:01:33.934954 (+176815us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.018893 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:34.020256 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:34.022704 14629 raft_consensus.cc:1226] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:34.020483 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:33.970844 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:33.975813 13657 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:34.025867 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:34.025995 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:34.026113 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 50403.8us
I20260807 08:01:34.026484 14625 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:34.027359 17010 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:34.027552 13657 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:34.027631 17179 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:34.027973 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:34.027894 17010 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.028260 17179 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.028936 13657 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:33.977905 13817 pending_rounds.cc:76] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 6
I20260807 08:01:34.029291 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:34.029664 13817 pending_rounds.cc:97] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260807 08:01:34.029981 13817 raft_consensus.cc:2866] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:34.030095 17010 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.030236 17179 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.917711 13811 rpcz_store.cc:276] 0807 08:01:31.622907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.912058 (+2289151us) service_pool.cc:227] Handling call
0807 08:01:33.912456 (+ 398us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.912677 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.912715 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:33.912772 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:33.912818 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.912883 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.913009 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.030584 17010 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:34.027631 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:34.031121 17010 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.032063 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:34.032748 13701 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:33.978097 13587 raft_consensus.cc:2396] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:34.032994 17010 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:33.978901 13812 raft_consensus.cc:3032] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:34.033739 13812 raft_consensus.cc:729] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:33.973299 14580 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:33.956670 17153 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:33.977550 13588 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:33.956430 17141 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:33.990353 13661 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:33.939560 13816 rpcz_store.cc:276] 0807 08:01:31.864163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.938283 (+2074120us) service_pool.cc:227] Handling call
0807 08:01:33.938629 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.938832 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.938866 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:33.938918 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:33.938961 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.939020 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:33.939135 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.005555 13705 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:34.034411 17010 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:34.037675 16992 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:34.038201 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:34.034183 13540 connection.cc:232] RPC call timeout handler was delayed by -2.4091778818983585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
I20260807 08:01:34.038319 16992 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:34.038976 13540 connection.cc:232] RPC call timeout handler was delayed by -1.8188354740850627s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
I20260807 08:01:34.035154 17153 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:34.039439 13540 connection.cc:232] RPC call timeout handler was delayed by -1.7990354741923511s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
I20260807 08:01:34.039628 17153 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:34.037618 17072 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:34.040037 13540 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:34.040408 16992 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.030634 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 413) took 3794 ms. Trace:
I20260807 08:01:34.042207 16661 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:34.042410 13817 rpcz_store.cc:276] 0807 08:01:30.236349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.897976 (+2661627us) service_pool.cc:227] Handling call
0807 08:01:32.986200 (+ 88224us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.030420 (+1044220us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.030588 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.037569 17048 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:34.041978 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:34.034969 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 63543.6us
I20260807 08:01:34.044462 17074 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.004938 13814 pending_rounds.cc:76] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 5
I20260807 08:01:34.045136 13814 pending_rounds.cc:97] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260807 08:01:34.045455 13814 raft_consensus.cc:2866] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:34.048440 16791 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:01:34.007515 13706 raft_consensus.cc:1226] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260807 08:01:34.050199 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1422) took 3840 ms. Trace:
I20260807 08:01:34.050603 13706 rpcz_store.cc:276] 0807 08:01:30.209550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.004677 (+3795127us) service_pool.cc:227] Handling call
0807 08:01:34.004968 (+ 291us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:34.049899 (+ 44931us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.050137 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.051605 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:34.052171 13706 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 2.
I20260807 08:01:34.052716 16459 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.008162 13585 raft_consensus.cc:1226] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 08:01:33.962579 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:34.009346 13537 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:34.094851 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1688) took 3334 ms. Trace:
W20260807 08:01:34.095149 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 85908.6us
I20260807 08:01:34.095311 13585 rpcz_store.cc:276] 0807 08:01:30.761007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.007619 (+3246612us) service_pool.cc:227] Handling call
0807 08:01:34.007954 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:34.094543 (+ 86589us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.094787 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:33.897212 17026 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.024622 17176 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:34.097223 13585 raft_consensus.cc:1226] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260807 08:01:34.024703 14629 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:34.044587 13701 raft_consensus.cc:2379] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:34.007766 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:34.035714 16893 raft_consensus.cc:2781] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 08:01:34.042011 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 51748.5us
I20260807 08:01:34.034574 13812 consensus_queue.cc:260] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.106189 13812 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:34.042740 16666 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:34.061236 16753 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:34.061929 16958 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:34.038931 13705 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:34.001329 13805 rpcz_store.cc:276] 0807 08:01:32.622335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.999993 (+1377658us) service_pool.cc:227] Handling call
0807 08:01:34.000291 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.000512 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.000548 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.000601 (+ 53us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:34.000642 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.000708 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.000842 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.175772 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:33.975174 17003 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.183314 17003 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:34.092031 17166 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.008005 17123 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:34.100644 17176 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.186801 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:34.186959 17123 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:34.187476 13701 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:34.066622 17218 consensus_queue.cc:260] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.101445 17127 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:34.067673 13700 raft_consensus.cc:2379] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:34.188906 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:34.189119 17127 raft_consensus.cc:686] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:34.190166 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:34.190603 13591 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:34.067905 13710 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:34.071245 17064 raft_consensus.cc:3032] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:34.191393 17064 raft_consensus.cc:729] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:34.174052 16401 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260807 08:01:34.192548 17064 consensus_queue.cc:260] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.038220 17010 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.080688 13817 raft_consensus.cc:3032] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:34.200466 13817 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:34.201468 13817 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.202703 13817 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:34.082305 13580 raft_consensus.cc:2396] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 08:01:34.085759 13702 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:34.086079 16816 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:33.952692 14618 rpcz_store.cc:276] 0807 08:01:30.797785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.783101 (+1985316us) service_pool.cc:227] Handling call
0807 08:01:33.721591 (+938490us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:33.827676 (+106085us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:33.837500 (+ 9824us) spinlock_profiling.cc:243] Waited 837 ms on lock 0x7b54006a04d0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:33.844984 (+ 7484us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1925430528}
I20260807 08:01:33.978160 17204 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:34.027355 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:34.090794 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.217271 13816 raft_consensus.cc:2396] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260807 08:01:33.997285 16976 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:34.188395 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:34.131129 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.226341 13814 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:34.227396 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.228684 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.146103 13812 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 08:01:34.169806 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:34.230002 13812 pending_rounds.cc:76] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 5
I20260807 08:01:34.230319 13812 pending_rounds.cc:97] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260807 08:01:34.230615 13812 raft_consensus.cc:2866] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:34.231379 13812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 423) took 1613 ms. Trace:
I20260807 08:01:34.231737 13812 rpcz_store.cc:276] 0807 08:01:32.620109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.977297 (+1357188us) service_pool.cc:227] Handling call
0807 08:01:33.978722 (+ 1425us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.231092 (+252370us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.231321 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1957120}
I20260807 08:01:34.232935 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:34.241544 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.243047 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.177064 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 335) took 1606 ms. Trace:
I20260807 08:01:34.177112 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:34.179147 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:34.181401 17026 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.238332 13661 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:34.180889 17153 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.159802 16893 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:34.182443 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 420) took 2104 ms. Trace:
I20260807 08:01:34.190063 17127 consensus_queue.cc:237] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.053118 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:34.263909 13706 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:34.216842 17218 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:34.213847 17065 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:34.195588 17062 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:34.223697 17064 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:34.220788 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.099792 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1690) took 3337 ms. Trace:
I20260807 08:01:34.273841 13585 rpcz_store.cc:276] 0807 08:01:30.762151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.096557 (+3334406us) service_pool.cc:227] Handling call
0807 08:01:34.096941 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:34.099503 (+ 2562us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.099727 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.244854 13812 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 6
I20260807 08:01:34.223222 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 329) took 1680 ms. Trace:
W20260807 08:01:34.277105 13432 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.028s sys 0.038s
I20260807 08:01:34.249207 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:34.277505 13432 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.525s user 0.028s sys 0.038s
I20260807 08:01:34.278369 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.246352 13709 rpcz_store.cc:276] 0807 08:01:32.571018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.911614 (+1340596us) service_pool.cc:227] Handling call
0807 08:01:34.176841 (+265227us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b5400321c54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:34.176975 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76582784}
I20260807 08:01:34.244526 13814 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:34.280128 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.280797 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.280874 13814 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 1.
I20260807 08:01:34.216138 16816 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:34.073628 17116 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:34.266387 13817 raft_consensus.cc:1261] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:01:34.291407 13812 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:34.268404 13705 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:34.219553 17204 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:34.319268 17153 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.213618 17141 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260807 08:01:34.256524 17157 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260807 08:01:34.282792 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:34.321574 17010 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.282009 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.321950 13585 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:34.274577 17065 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.322942 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:34.323529 13585 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
W20260807 08:01:34.283190 14583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.015s sys 0.045s
I20260807 08:01:34.324370 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:34.324638 14583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.587s user 0.015s sys 0.046s
I20260807 08:01:34.325088 13585 raft_consensus.cc:2379] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 6: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:34.255609 17154 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:34.325583 17154 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.325919 17005 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.326396 13705 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:34.327014 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 328) took 1787 ms. Trace:
I20260807 08:01:34.327706 17204 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.328579 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:34.234205 17176 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.286645 13805 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:34.263356 16893 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:34.282747 13702 rpcz_store.cc:276] 0807 08:01:32.539427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.907602 (+1368175us) service_pool.cc:227] Handling call
0807 08:01:34.220028 (+312426us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.300957 17048 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:34.275759 17062 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.276059 17074 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:34.293161 17116 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:34.330305 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:34.247704 17112 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:34.328982 13700 raft_consensus.cc:2379] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:34.346737 16816 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.347951 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:34.247653 17025 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.263502 13808 rpcz_store.cc:276] 0807 08:01:32.078485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.951131 (+1872646us) service_pool.cc:227] Handling call
0807 08:01:34.006360 (+ 55229us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.087473 (+ 81113us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.098021 (+ 10548us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.101179 (+ 3158us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:34.160346 (+ 59167us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.173631 (+ 13285us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.182377 (+ 8746us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.352514 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:34.269382 16679 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260807 08:01:34.357009 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:34.359393 17125 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.015s sys 0.026s
I20260807 08:01:34.327634 17154 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.282084 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.369732 13585 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:34.330603 13584 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:34.339174 16551 consensus_meta.cc:287] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.511s user 0.011s sys 0.002s
I20260807 08:01:34.330380 17204 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.418581 13590 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:34.324380 17065 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:34.400414 13769 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.008s sys 0.006s
W20260807 08:01:34.426684 13769 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.555s user 0.008s sys 0.006s
W20260807 08:01:34.427069 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 555864us
I20260807 08:01:34.428172 13769 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:34.429097 13769 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:34.431622 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1691) took 3669 ms. Trace:
I20260807 08:01:34.432083 13591 rpcz_store.cc:276] 0807 08:01:30.762480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.101748 (+3339268us) service_pool.cc:227] Handling call
0807 08:01:34.178803 (+ 77055us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.191017 (+ 12214us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.431330 (+240313us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b540021f7d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:34.431552 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26683776}
I20260807 08:01:34.411567 13585 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 08:01:34.327841 17005 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:34.400071 13809 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.005s sys 0.008s
W20260807 08:01:34.445689 13809 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.612s user 0.005s sys 0.008s
I20260807 08:01:34.446790 13769 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:34.449288 13769 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:34.434134 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 435) took 2280 ms. Trace:
W20260807 08:01:34.426499 13710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1714) took 4327 ms (client timeout 1655 ms). Trace:
I20260807 08:01:34.339438 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 332) took 1770 ms. Trace:
I20260807 08:01:34.386508 17112 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.330955 16927 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.420189 16459 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.435732 17196 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:34.333045 17123 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:34.450762 13710 rpcz_store.cc:269] 0807 08:01:30.099271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.099449 (+ 178us) service_pool.cc:227] Handling call
0807 08:01:34.426289 (+4326840us) spinlock_profiling.cc:243] Waited 3.83 s on lock 0x7b54001dc9d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:34.426419 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8805555968}
W20260807 08:01:34.416253 17125 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.015s sys 0.027s
W20260807 08:01:34.488765 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 151055us
I20260807 08:01:34.384377 16897 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.492676 13710 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 08:01:34.457362 17066 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:34.497270 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 333) took 1927 ms. Trace:
I20260807 08:01:34.499001 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:34.499284 13710 rpcz_store.cc:276] 0807 08:01:32.570321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.492016 (+1921695us) service_pool.cc:227] Handling call
0807 08:01:34.492349 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.497008 (+ 4659us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.497211 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.499728 13814 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:34.457809 17159 raft_consensus.cc:2781] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:01:34.457706 16958 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:34.511283 13814 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:34.509336 14583 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:34.516392 14583 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:34.517606 14583 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:34.516085 14580 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:34.528136 14583 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:34.528930 16753 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:34.530845 17005 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:34.531261 17005 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:34.531585 17005 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:34.538196 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:34.539322 16753 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:34.541579 14583 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:34.503588 17066 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:34.555159 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:34.542814 16679 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:34.526319 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:34.556274 13805 raft_consensus.cc:2354] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:34.671422 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.671908 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.672569 13808 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:34.673694 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:34.673316 13805 raft_consensus.cc:2379] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:34.675516 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.450582 13585 rpcz_store.cc:276] 0807 08:01:32.153858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.410799 (+2256941us) service_pool.cc:227] Handling call
0807 08:01:34.411162 (+ 363us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.433810 (+ 22648us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.434063 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.684867 13657 leader_election.cc:398] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:34.685456 13657 leader_election.cc:401] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260807 08:01:34.631184 13587 raft_consensus.cc:1226] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:34.688756 16661 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:34.548210 17066 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:34.633493 13592 raft_consensus.cc:2354] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
W20260807 08:01:34.648231 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 409849us
I20260807 08:01:34.734889 13661 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:34.736574 13592 raft_consensus.cc:1261] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:01:34.689038 17062 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.737648 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1719) took 1923 ms. Trace:
I20260807 08:01:34.738198 13592 rpcz_store.cc:276] 0807 08:01:32.813948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.732207 (+1918259us) service_pool.cc:227] Handling call
0807 08:01:34.732934 (+ 727us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.737222 (+ 4288us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.737544 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.452248 13702 rpcz_store.cc:276] 0807 08:01:32.568640 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.337958 (+1769318us) service_pool.cc:227] Handling call
0807 08:01:34.338536 (+ 578us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.338817 (+ 281us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.338876 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.338964 (+ 88us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:34.339029 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.339123 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.339308 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.741341 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1720) took 1914 ms. Trace:
I20260807 08:01:34.327308 13705 rpcz_store.cc:276] 0807 08:01:32.539127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.325831 (+1786704us) service_pool.cc:227] Handling call
0807 08:01:34.326087 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.326781 (+ 694us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.326959 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.653038 13703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1421) took 4443 ms. Trace:
I20260807 08:01:34.773934 13703 rpcz_store.cc:276] 0807 08:01:30.209158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.002120 (+3792962us) service_pool.cc:227] Handling call
0807 08:01:34.088757 (+ 86637us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:34.089019 (+ 262us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.089053 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:34.098654 (+ 9601us) log.cc:842] Serialized 26 byte log entry
0807 08:01:34.650641 (+551987us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:34.651500 (+ 859us) log.cc:842] Serialized 14 byte log entry
0807 08:01:34.652097 (+ 597us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.652314 (+ 217us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:34.652418 (+ 104us) raft_consensus.cc:1671] finished
0807 08:01:34.652440 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.652740 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72882,"prepare.queue_time_us":46449,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":33237,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":345,"thread_start_us":471719,"threads_started":1,"wal-append.queue_time_us":472704}
I20260807 08:01:34.668380 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.777705 17123 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.778725 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:34.784986 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 732814us
I20260807 08:01:34.676103 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:34.799746 13808 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:34.628561 13537 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:34.802245 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 174056us
I20260807 08:01:34.803268 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54058 (request call id 1305) took 1022 ms. Trace:
I20260807 08:01:34.803900 13537 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:34.803845 13811 rpcz_store.cc:276] 0807 08:01:33.780529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.032662 (+252133us) service_pool.cc:227] Handling call
0807 08:01:34.740851 (+708189us) spinlock_profiling.cc:243] Waited 6.79 ms on lock 0x7b54001fdbd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0af0 00007f6b1cf73f2a 00007f6b21eb0c6e 00007f6b1b5a7807 00007f6b1b5a7792 00007f6b1b5a76c1 00007f6b1b5a763c 00007f6b1b5a68f2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:34.802932 (+ 62081us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15616896}
I20260807 08:01:34.559513 14583 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:34.678335 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.806099 13812 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:34.806212 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:34.807165 13814 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 1 FOLLOWER]: Advancing to term 3
I20260807 08:01:34.564145 16679 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:34.810994 16679 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:34.811801 13540 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:34.548826 16753 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:34.550920 16816 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.565927 16958 raft_consensus.cc:686] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:01:34.836776 17218 consensus_meta.cc:287] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.565s user 0.014s sys 0.003s
I20260807 08:01:34.836194 16958 consensus_queue.cc:237] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.842561 17064 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:34.843104 17064 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:34.843499 17064 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:34.581346 16927 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.550678 17025 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.581730 17112 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.575917 17157 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:34.554637 17190 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.882823 16753 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:34.885190 16992 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:34.885782 16992 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:34.886158 16992 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:34.886634 17116 consensus_meta.cc:287] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.544s user 0.011s sys 0.012s
I20260807 08:01:34.547366 17146 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:34.895629 17146 raft_consensus.cc:686] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:34.896472 17005 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.480587 17196 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.576790 17215 raft_consensus.cc:3032] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Stepping down as leader of term 4
W20260807 08:01:34.589280 17226 consensus_meta.cc:287] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.602s user 0.014s sys 0.000s
I20260807 08:01:34.905081 17215 raft_consensus.cc:729] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:34.905448 17226 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.907155 17215 consensus_queue.cc:260] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.631913 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1713) took 1938 ms. Trace:
I20260807 08:01:34.716867 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1712) took 2026 ms. Trace:
W20260807 08:01:34.717859 13580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1686) took 3957 ms (client timeout 2052 ms). Trace:
W20260807 08:01:34.909693 13580 rpcz_store.cc:269] 0807 08:01:30.760241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.756733 (+1996492us) service_pool.cc:227] Handling call
0807 08:01:34.717200 (+1960467us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400164ed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:34.717504 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2741938048}
W20260807 08:01:34.720278 13590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1715) took 1952 ms (client timeout 2053 ms). Trace:
W20260807 08:01:34.720610 16863 consensus_meta.cc:287] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.749s user 0.002s sys 0.012s
I20260807 08:01:34.720775 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:34.723291 16897 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.584385 16893 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.915498 13537 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:34.915935 16863 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.917707 16893 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.918040 13711 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:34.584735 16976 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.728870 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1419) took 4520 ms. Trace:
W20260807 08:01:34.911221 13590 rpcz_store.cc:269] 0807 08:01:32.768105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.198870 (+430765us) service_pool.cc:227] Handling call
0807 08:01:34.719754 (+1520884us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b5400077454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:34.720022 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1641333120}
I20260807 08:01:34.911700 13587 rpcz_store.cc:276] 0807 08:01:32.690989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.411775 (+1720786us) service_pool.cc:227] Handling call
0807 08:01:34.412142 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.688231 (+276089us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.716335 (+ 28104us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b5400076f54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:34.716747 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":502242688}
I20260807 08:01:34.920107 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 346) took 1124 ms. Trace:
I20260807 08:01:34.918699 13708 rpcz_store.cc:276] 0807 08:01:30.208395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:32.511817 (+2303422us) service_pool.cc:227] Handling call
0807 08:01:32.558105 (+ 46288us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:33.219968 (+661863us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:33.220013 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:33.702150 (+482137us) log.cc:842] Serialized 26 byte log entry
0807 08:01:34.009981 (+307831us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:34.211335 (+201354us) log.cc:842] Serialized 14 byte log entry
0807 08:01:34.481085 (+269750us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.728070 (+246985us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b5400180954. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:34.728143 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:34.728211 (+ 68us) raft_consensus.cc:1671] finished
0807 08:01:34.728341 (+ 130us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.728645 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":166335,"prepare.queue_time_us":201044,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":37,"raft.queue_time_us":65133,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":7538,"spinlock_wait_cycles":1505692544,"thread_start_us":517861,"threads_started":2,"wal-append.queue_time_us":593382}
I20260807 08:01:34.921732 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 440) took 1538 ms. Trace:
I20260807 08:01:34.920552 13705 rpcz_store.cc:276] 0807 08:01:33.795140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.771182 (+976042us) service_pool.cc:227] Handling call
0807 08:01:34.771771 (+ 589us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:34.797943 (+ 26172us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.797979 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:34.798567 (+ 588us) log.cc:842] Serialized 26 byte log entry
0807 08:01:34.853227 (+ 54660us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:34.853276 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.853380 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:34.919765 (+ 66385us) raft_consensus.cc:1671] finished
0807 08:01:34.919791 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.919925 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25409,"raft.queue_time_us":30217,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":124,"thread_start_us":54177,"threads_started":1,"wal-append.queue_time_us":88846}
I20260807 08:01:34.767843 17267 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:34.923134 17267 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:34.668917 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 219498us
I20260807 08:01:34.929145 16661 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) with higher term. Message: Invalid argument: T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:34.922246 13587 rpcz_store.cc:276] 0807 08:01:33.382901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.920613 (+1537712us) service_pool.cc:227] Handling call
0807 08:01:34.921035 (+ 422us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.921344 (+ 309us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.921375 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.921410 (+ 35us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:34.921479 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.921559 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.921675 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.887205 17116 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.931507 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 442) took 1226 ms. Trace:
W20260807 08:01:34.787890 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 53338.7us
I20260807 08:01:34.932307 17116 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.934602 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:34.936303 13661 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:34.789140 17258 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:34.792258 13589 raft_consensus.cc:1226] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:34.794431 13584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1731) took 1045 ms. Trace:
W20260807 08:01:34.864387 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 52795.9us
I20260807 08:01:34.877166 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1714) took 2068 ms. Trace:
I20260807 08:01:34.741861 13592 rpcz_store.cc:276] 0807 08:01:32.826634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.739558 (+1912924us) service_pool.cc:227] Handling call
0807 08:01:34.740181 (+ 623us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.740732 (+ 551us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.740781 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.740874 (+ 93us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:34.740941 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.741056 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.741233 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.908236 17226 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:34.543051 17242 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:34.908896 13591 rpcz_store.cc:276] 0807 08:01:32.693402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.433132 (+1739730us) service_pool.cc:227] Handling call
0807 08:01:34.630761 (+197629us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.631258 (+ 497us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.631303 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.631361 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:34.631401 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.631470 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.631656 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.914949 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1424) took 4507 ms. Trace:
I20260807 08:01:34.994724 13706 rpcz_store.cc:276] 0807 08:01:30.396886 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.264485 (+3867599us) service_pool.cc:227] Handling call
0807 08:01:34.264793 (+ 308us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:34.492362 (+227569us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.492394 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:34.502894 (+ 10500us) log.cc:842] Serialized 26 byte log entry
0807 08:01:34.698357 (+195463us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:34.816197 (+117840us) log.cc:842] Serialized 14 byte log entry
0807 08:01:34.818527 (+ 2330us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.875047 (+ 56520us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54002d50d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:34.875114 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:34.877051 (+ 1937us) raft_consensus.cc:1671] finished
0807 08:01:34.891518 (+ 14467us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.904198 (+ 12680us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16173,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":5012,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":18363,"spinlock_wait_cycles":522499840,"thread_start_us":163250,"threads_started":2,"wal-append.queue_time_us":187146}
I20260807 08:01:34.915273 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 437) took 1717 ms. Trace:
I20260807 08:01:34.997870 13580 rpcz_store.cc:276] 0807 08:01:33.197792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.910806 (+1713014us) service_pool.cc:227] Handling call
0807 08:01:34.912979 (+ 2173us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.914915 (+ 1936us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.914955 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.915000 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:34.915042 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.915094 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.915208 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.921850 16897 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:35.000458 16897 raft_consensus.cc:686] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:35.000854 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 326) took 2462 ms. Trace:
I20260807 08:01:34.899101 13814 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:34.940868 17123 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:35.002977 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 64468.3us
I20260807 08:01:35.004109 13769 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:34.937382 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1721) took 2094 ms. Trace:
I20260807 08:01:35.005673 13589 rpcz_store.cc:276] 0807 08:01:32.842602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.791271 (+1948669us) service_pool.cc:227] Handling call
0807 08:01:34.791800 (+ 529us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.937135 (+145335us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.937327 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:35.006552 13769 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:35.007776 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 451) took 1300 ms. Trace:
I20260807 08:01:34.937561 13584 rpcz_store.cc:276] 0807 08:01:33.749156 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.898574 (+149418us) service_pool.cc:227] Handling call
0807 08:01:34.793970 (+895396us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54001649d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:34.794197 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":836750720}
I20260807 08:01:35.008605 17294 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:35.008790 17157 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:34.937805 13585 rpcz_store.cc:276] 0807 08:01:32.696644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.684569 (+1987925us) service_pool.cc:227] Handling call
0807 08:01:34.685377 (+ 808us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:34.686273 (+ 896us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.686326 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:34.687293 (+ 967us) log.cc:842] Serialized 26 byte log entry
0807 08:01:34.761512 (+ 74219us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:34.762986 (+ 1474us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.763058 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:34.763095 (+ 37us) raft_consensus.cc:1671] finished
0807 08:01:34.763105 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.764057 (+ 952us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2095,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":255,"thread_start_us":73676,"threads_started":1,"wal-append.queue_time_us":66679}
I20260807 08:01:35.009613 17294 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:34.953332 17196 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.010749 17159 raft_consensus.cc:686] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:35.008232 13589 rpcz_store.cc:276] 0807 08:01:33.707714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:35.006572 (+1298858us) service_pool.cc:227] Handling call
0807 08:01:35.007021 (+ 449us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:35.007305 (+ 284us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:35.007337 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:35.007391 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:35.007487 (+ 96us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:35.007574 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:35.007705 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:35.011766 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 339) took 1236 ms. Trace:
I20260807 08:01:35.012255 17159 consensus_queue.cc:237] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.013684 13584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 453) took 1305 ms. Trace:
I20260807 08:01:35.014145 13584 rpcz_store.cc:276] 0807 08:01:33.708380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:35.009187 (+1300807us) service_pool.cc:227] Handling call
0807 08:01:35.013062 (+ 3875us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:35.013321 (+ 259us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:35.013346 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:35.013409 (+ 63us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:35.013451 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:35.013502 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:35.013624 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.920223 16976 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:35.015715 16897 consensus_queue.cc:237] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.012197 13711 rpcz_store.cc:276] 0807 08:01:33.775580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.952333 (+176753us) service_pool.cc:227] Handling call
0807 08:01:35.011461 (+1059128us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b5400082fd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:35.011680 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2052056960}
I20260807 08:01:35.017593 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 446) took 1311 ms. Trace:
I20260807 08:01:35.018097 13592 rpcz_store.cc:276] 0807 08:01:33.706124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:35.016373 (+1310249us) service_pool.cc:227] Handling call
0807 08:01:35.016833 (+ 460us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:35.017126 (+ 293us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:35.017152 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:35.017190 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:35.017230 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:35.017279 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:35.017501 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.944128 13540 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:34.947260 16661 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:34.946331 13537 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:34.946439 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 352682us
I20260807 08:01:35.020820 16661 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:34.948748 13661 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:35.021425 16661 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:35.022245 16976 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 08:01:35.022504 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 73928.4us
I20260807 08:01:35.022737 16976 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:35.023252 16976 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:35.023432 13661 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.024569 17141 raft_consensus.cc:2781] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:35.024959 13661 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.025173 17141 raft_consensus.cc:686] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:34.956060 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1726) took 4537 ms. Trace:
W20260807 08:01:34.957055 17304 log.cc:924] Time spent T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Append to log took a long time: real 0.058s user 0.000s sys 0.002s
W20260807 08:01:35.026412 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 82396.4us
I20260807 08:01:35.027168 17226 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:01:35.027549 13540 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:34.969919 16863 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.012369 13585 raft_consensus.cc:1261] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 08:01:34.972755 17066 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:34.920126 13590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 438) took 1711 ms. Trace:
I20260807 08:01:35.033712 13590 rpcz_store.cc:276] 0807 08:01:33.208793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.919045 (+1710252us) service_pool.cc:227] Handling call
0807 08:01:34.919499 (+ 454us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.919799 (+ 300us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.919829 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.919867 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:34.919904 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.919955 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.920068 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.983810 13579 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
W20260807 08:01:34.985208 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 86113.5us
I20260807 08:01:34.925458 17267 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:34.896981 17146 consensus_queue.cc:237] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:34.909341 17215 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:35.005199 17298 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:35.009914 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 324) took 2484 ms. Trace:
W20260807 08:01:35.026496 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 80290.1us
I20260807 08:01:34.946146 17005 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:34.931883 13587 rpcz_store.cc:276] 0807 08:01:33.704970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.930470 (+1225500us) service_pool.cc:227] Handling call
0807 08:01:34.930880 (+ 410us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.931185 (+ 305us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.931212 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.931248 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:34.931282 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.931346 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.931460 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:34.975843 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:35.021569 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:35.035602 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 448) took 1322 ms. Trace:
I20260807 08:01:35.057497 13808 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:35.009160 13769 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:35.059824 13580 rpcz_store.cc:276] 0807 08:01:33.706766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.998937 (+1292171us) service_pool.cc:227] Handling call
0807 08:01:34.999361 (+ 424us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.999608 (+ 247us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.999638 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.999701 (+ 63us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:34.999753 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.999816 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.999929 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:35.027626 17226 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:35.067709 17226 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:35.063892 13657 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:35.026037 13809 rpcz_store.cc:276] 0807 08:01:30.314057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:33.820251 (+3506194us) service_pool.cc:227] Handling call
0807 08:01:33.820567 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:33.820755 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 08:01:34.446125 (+625370us) log.cc:842] Serialized 14 byte log entry
0807 08:01:34.761929 (+315804us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:34.766676 (+ 4747us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:34.823561 (+ 56885us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.828894 (+ 5333us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:34.851815 (+ 22921us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30169,"prepare.queue_time_us":673225,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":20592,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"thread_start_us":927248,"threads_started":3,"wal-append.queue_time_us":139780}
I20260807 08:01:35.074792 17190 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:35.074903 17116 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:35.033131 17196 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:35.069139 16679 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.075904 16992 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.088107 16679 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:35.088455 17303 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:35.088677 16992 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:35.089010 17303 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:35.089468 17303 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:35.090575 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 65771.6us
I20260807 08:01:35.091502 13661 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:35.094300 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 56731.9us
W20260807 08:01:35.094846 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 67486us
I20260807 08:01:35.059046 13709 rpcz_store.cc:276] 0807 08:01:32.525114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.279523 (+1754409us) service_pool.cc:227] Handling call
0807 08:01:34.297556 (+ 18033us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:34.297979 (+ 423us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.298014 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:34.387339 (+ 89325us) log.cc:842] Serialized 52 byte log entry
0807 08:01:34.844637 (+457298us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:34.845210 (+ 573us) log.cc:842] Serialized 14 byte log entry
0807 08:01:34.852136 (+ 6926us) log.cc:842] Serialized 14 byte log entry
0807 08:01:34.857054 (+ 4918us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:34.868473 (+ 11419us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:34.934612 (+ 66139us) spinlock_profiling.cc:243] Waited 8.87 ms on lock 0x7b18003b8540. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043764 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:34.940141 (+ 5529us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b18003b8540. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:34.940672 (+ 531us) spinlock_profiling.cc:243] Waited 5.52 ms on lock 0x7b18003b8540. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a010f96 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a010f96 00007f6b1a010f30
0807 08:01:34.942187 (+ 1515us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b18003b8540. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a010f96 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b21e127f1 00007f6b21e098e7 00007f6b21e0977a 00007f6b21efd95f
0807 08:01:34.945453 (+ 3266us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b4800061d58. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b21e127f1 00007f6b21e098e7 00007f6b21e0977a 00007f6b21efd95f 00007f6b21efd627 00007f6b21f11b2c 00007f6b21f11ad8 00007f6b21f11a58 00007f6b21f11a13 00007f6b21f10cbc 00007f6b20e92be6
0807 08:01:35.009478 (+ 64025us) raft_consensus.cc:1671] finished
0807 08:01:35.009505 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:35.009717 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46269,"prepare.queue_time_us":83707,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":93,"raft.queue_time_us":53908,"raft.run_cpu_time_us":4959,"raft.run_wall_time_us":100726,"spinlock_wait_cycles":52015744,"thread_start_us":390214,"threads_started":1,"wal-append.queue_time_us":465320}
I20260807 08:01:35.081542 17196 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:35.012599 13701 rpcz_store.cc:276] 0807 08:01:32.538377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.293115 (+1754738us) service_pool.cc:227] Handling call
0807 08:01:34.317989 (+ 24874us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:34.358464 (+ 40475us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.384179 (+ 25715us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:34.495937 (+111758us) log.cc:842] Serialized 26 byte log entry
0807 08:01:34.665507 (+169570us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:34.714449 (+ 48942us) log.cc:842] Serialized 14 byte log entry
0807 08:01:35.000037 (+285588us) spinlock_profiling.cc:243] Waited 81.1 ms on lock 0x7b54000fc6a0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cefefb3 00007f6b1cf6bf0f 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c
0807 08:01:35.000119 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:35.000287 (+ 168us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:35.000352 (+ 65us) raft_consensus.cc:1671] finished
0807 08:01:35.000460 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:35.000649 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122477,"prepare.queue_time_us":44395,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":154497,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":10683,"spinlock_wait_cycles":186527616,"thread_start_us":59320,"threads_started":2,"wal-append.queue_time_us":161022}
I20260807 08:01:35.094440 17298 raft_consensus.cc:2726] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:35.097671 17218 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:35.026422 17141 consensus_queue.cc:237] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.110507 17196 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:35.112031 16863 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election won for term 3
W20260807 08:01:35.132161 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 110799us
I20260807 08:01:35.118412 17190 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:35.141166 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:35.145018 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 447) took 1438 ms. Trace:
W20260807 08:01:35.147344 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 138323us
I20260807 08:01:35.145859 13591 rpcz_store.cc:276] 0807 08:01:33.706431 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.994739 (+1288308us) service_pool.cc:227] Handling call
0807 08:01:34.995271 (+ 532us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:34.995819 (+ 548us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.995849 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:34.996461 (+ 612us) log.cc:842] Serialized 52 byte log entry
0807 08:01:35.113178 (+116717us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:35.113232 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:35.113328 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:35.144745 (+ 31417us) raft_consensus.cc:1671] finished
0807 08:01:35.144775 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:35.144953 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4841,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"thread_start_us":116227,"threads_started":1,"wal-append.queue_time_us":145989}
I20260807 08:01:35.153601 13540 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:35.148185 13537 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:35.148770 16863 raft_consensus.cc:686] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:35.180414 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 48807.4us
I20260807 08:01:35.181795 16816 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:35.182554 16816 raft_consensus.cc:686] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:35.183568 16816 consensus_queue.cc:237] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:35.184572 13579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1683) took 4491 ms (client timeout 1791 ms). Trace:
W20260807 08:01:35.185019 13579 rpcz_store.cc:269] 0807 08:01:30.692899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:30.821681 (+128782us) service_pool.cc:227] Handling call
0807 08:01:32.622113 (+1800432us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b5800001898. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:35.184316 (+2562203us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x7b54001360d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:35.184452 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5203641472}
I20260807 08:01:35.185873 17256 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:35.189270 17218 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:35.189924 17218 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:35.198467 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 82041.5us
I20260807 08:01:35.200623 13588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 434) took 2943 ms. Trace:
W20260807 08:01:35.203277 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 52633.2us
I20260807 08:01:35.203882 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 334) took 2534 ms. Trace:
I20260807 08:01:35.205875 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:35.209450 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:35.211174 13808 raft_consensus.cc:2379] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:35.211392 13769 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:35.202570 17066 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:35.218325 13661 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.217383 17298 raft_consensus.cc:2741] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
W20260807 08:01:35.224159 17324 log.cc:924] Time spent T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260807 08:01:35.206842 13710 rpcz_store.cc:276] 0807 08:01:32.570665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.500263 (+1929598us) service_pool.cc:227] Handling call
0807 08:01:34.509405 (+ 9142us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:34.509934 (+ 529us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:34.509980 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:34.510518 (+ 538us) log.cc:842] Serialized 26 byte log entry
0807 08:01:35.005197 (+494679us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b180041c6e0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:35.057146 (+ 51949us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:35.071087 (+ 13941us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:35.077591 (+ 6504us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:35.077658 (+ 67us) raft_consensus.cc:1671] finished
0807 08:01:35.077682 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:35.105005 (+ 27323us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1330,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":31015,"spinlock_wait_cycles":6587776,"thread_start_us":364891,"threads_started":1,"wal-append.queue_time_us":372680}
I20260807 08:01:35.202963 13588 rpcz_store.cc:276] 0807 08:01:32.153190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:34.387791 (+2234601us) service_pool.cc:227] Handling call
0807 08:01:34.419003 (+ 31212us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:34.419239 (+ 236us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 08:01:34.860171 (+440932us) log.cc:842] Serialized 14 byte log entry
0807 08:01:35.081831 (+221660us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:35.093084 (+ 11253us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:35.093230 (+ 146us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:35.093432 (+ 202us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:35.096672 (+ 3240us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13219,"prepare.queue_time_us":144434,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":883,"raft.queue_time_us":26339,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"spinlock_wait_cycles":61312,"thread_start_us":537078,"threads_started":2,"wal-append.queue_time_us":259860}
I20260807 08:01:35.226039 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:35.238305 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:35.244140 17256 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:35.244669 17256 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:35.245147 17256 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:35.252233 17320 log.cc:924] Time spent T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20260807 08:01:35.254796 13540 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:35.258505 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 78908.7us
I20260807 08:01:35.224983 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:35.283922 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 79027.1us
I20260807 08:01:35.283711 16976 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:35.284713 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 78445.3us
I20260807 08:01:35.264618 17190 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:35.279439 16863 consensus_queue.cc:237] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.289813 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:35.271267 16992 raft_consensus.cc:2726] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:35.298576 16661 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.302678 17303 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.313649 13769 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:35.301290 16661 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.243018 17064 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.339165 17315 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:35.339663 17315 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:35.340039 17315 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:35.351363 13661 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.352447 17062 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:35.352885 13661 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:35.350667 17123 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:35.330550 17258 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:35.334080 17159 raft_consensus.cc:2726] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:35.378988 14583 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.366070 17062 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260807 08:01:35.414031 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 122509us
I20260807 08:01:35.385438 16679 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 08:01:35.419775 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 59825.2us
I20260807 08:01:35.420725 14583 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:35.421696 16893 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:35.422124 16893 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:35.422529 16679 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:35.422550 16893 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:01:35.426158 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 65947.7us
I20260807 08:01:35.386992 17062 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:35.433316 17062 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:35.455132 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:35.455364 13769 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:35.457533 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 59416.1us
I20260807 08:01:35.464035 17242 consensus_queue.cc:1035] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:35.456120 16976 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.479297 17064 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:35.484133 17159 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 08:01:35.494052 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 55972.6us
I20260807 08:01:35.495038 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:35.496752 13769 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:35.497680 16958 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:35.497812 17215 raft_consensus.cc:2726] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:35.526057 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:35.498065 16958 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:35.525782 17303 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:35.557246 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 94273us
W20260807 08:01:35.557590 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 60963.1us
I20260807 08:01:35.547937 16958 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:35.547508 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:35.549254 16976 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 08:01:35.572585 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 83805.7us
I20260807 08:01:35.582639 13661 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.584069 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:35.584525 17256 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.584967 13661 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:35.586472 16976 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:35.585438 17342 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:35.588979 17315 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.586198 13661 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.591687 17258 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:35.591954 17315 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:35.587683 17141 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:35.592597 17141 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:35.592984 17141 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:35.593873 13661 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.595196 17258 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:35.575403 13804 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:35.558588 16893 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.579447 16816 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:35.601055 17215 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:35.601315 17346 log.cc:924] Time spent T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260807 08:01:35.610643 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:35.610904 16893 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:35.611102 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:35.596151 16976 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:35.617355 13769 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:35.617612 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:35.619184 17218 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:35.620239 17226 consensus_meta.cc:287] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.549s user 0.005s sys 0.006s
I20260807 08:01:35.620673 17226 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.622704 17196 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.621249 17218 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.621676 13661 leader_election.cc:398] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:35.623824 13661 leader_election.cc:401] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:35.624606 17342 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 2 FOLLOWER]: Advancing to term 4
I20260807 08:01:35.626178 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:35.607333 17215 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.629321 17215 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:35.625012 17196 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:35.630355 17226 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.632020 17342 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4.
I20260807 08:01:35.636511 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:35.634325 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:35.632921 13701 raft_consensus.cc:1226] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260807 08:01:35.646399 13769 leader_election.cc:334] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Remote error: Timed out: Call waited in the queue past client deadline
I20260807 08:01:35.633301 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:35.608516 16816 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.663772 14626 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:35.681402 13711 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:35.666160 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:35.648509 16958 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.669351 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:35.671852 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:35.683189 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:35.684056 14627 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:35.655695 17062 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.642006 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:35.652961 17256 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.674284 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:35.675724 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:35.614528 16976 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:35.686160 13700 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:35.686615 14623 raft_consensus.cc:2379] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:35.652045 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:35.687750 14629 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:35.667637 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:35.689311 13703 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:35.676867 16661 consensus_queue.cc:1046] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260807 08:01:35.689455 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:35.690800 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:35.676502 17258 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.016s
I20260807 08:01:35.692205 17196 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:35.665103 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:35.693624 17196 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.694106 14618 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:35.646677 17159 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:35.698019 17123 raft_consensus.cc:3032] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:35.698489 17123 raft_consensus.cc:729] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:35.699421 17123 consensus_queue.cc:260] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.704344 13808 raft_consensus.cc:3032] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:35.704800 13808 raft_consensus.cc:729] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:35.705749 13808 consensus_queue.cc:260] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.706976 13808 raft_consensus.cc:3037] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:35.693215 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:35.711731 13701 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:35.713866 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:35.714673 13588 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:35.714941 17258 consensus_queue.cc:1046] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:35.716666 16863 raft_consensus.cc:3032] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:35.717000 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:35.717154 16863 raft_consensus.cc:729] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:35.717839 13588 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:35.672111 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:35.718706 16863 consensus_queue.cc:260] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.720360 16863 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:35.726929 17141 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.744393 17141 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.753008 16816 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:35.754196 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 107844us
I20260807 08:01:35.755591 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:35.756601 17159 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:35.758546 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:35.758800 17062 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.759192 13812 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:35.714466 13661 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:35.760334 13537 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:35.764097 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:35.761551 17190 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:35.765540 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:35.774551 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:35.775159 13812 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:35.767403 13703 raft_consensus.cc:1261] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 08:01:35.776363 13816 raft_consensus.cc:1261] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:35.768613 16976 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.770860 13710 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:35.781942 16976 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.784503 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:35.765839 17190 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:35.785780 17123 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:35.785953 17190 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:35.787281 13587 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:35.787878 13701 raft_consensus.cc:1261] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:35.789347 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:35.790138 13590 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:35.791966 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:35.792609 13584 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:35.793021 13700 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:35.769541 17196 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:35.796707 13537 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:35.798343 16992 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:35.798738 16992 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:35.797377 13708 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:35.801970 13710 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:35.806036 13588 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:35.818145 13808 raft_consensus.cc:1261] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 08:01:35.818164 13584 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:35.798858 16976 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:35.831136 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 59777.6us
I20260807 08:01:35.799149 16958 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.764438 16816 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:35.799134 16992 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:35.848898 17048 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
W20260807 08:01:35.852366 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 138007us
I20260807 08:01:35.865372 13808 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:35.843299 16976 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.868069 16976 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.868520 17242 consensus_queue.cc:1046] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:01:35.819295 17218 consensus_queue.cc:1035] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260807 08:01:35.868700 16976 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:35.838527 13804 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:35.842922 16816 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:35.883839 13540 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:35.895381 13584 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:35.894681 13816 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:35.894665 13432 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56582 (request call id 305) took 8751 ms. Trace:
I20260807 08:01:35.918279 13428 catalog_manager.cc:5495] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:35.880329 17242 raft_consensus.cc:3032] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:35.895056 16893 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:35.946018 16893 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.948213 16893 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:35.938045 13537 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:35.950484 17218 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:35.950979 17218 raft_consensus.cc:686] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:35.951237 16791 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260807 08:01:35.951895 17218 consensus_queue.cc:237] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:35.949882 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:35.956327 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 72597.2us
I20260807 08:01:35.884714 17062 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:35.958177 17062 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.958662 13540 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.959561 17196 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:35.960199 17062 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:35.915557 16816 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:35.926810 13428 catalog_manager.cc:5495] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:35.961642 13428 catalog_manager.cc:5495] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 1 to 4, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 4 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:35.963521 14618 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:35.963608 13428 catalog_manager.cc:5495] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:35.942551 17190 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.914453 13432 rpcz_store.cc:276] 0807 08:01:27.174044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:27.182069 (+ 8025us) service_pool.cc:227] Handling call
0807 08:01:35.894407 (+8712338us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:34.437357 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:34.557617 (+120260us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:34.559199 (+ 1582us) write_op.cc:435] Acquired schema lock
0807 08:01:34.561247 (+ 2048us) tablet.cc:660] Decoding operations
0807 08:01:34.722172 (+160925us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:34.722383 (+ 211us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:34.722433 (+ 50us) tablet.cc:683] Acquiring locks for 8 operations
0807 08:01:34.724092 (+ 1659us) tablet.cc:699] Row locks acquired
0807 08:01:34.724145 (+ 53us) write_op.cc:260] PREPARE: finished
0807 08:01:34.724508 (+ 363us) write_op.cc:270] Start()
0807 08:01:34.724808 (+ 300us) write_op.cc:276] Timestamp: P: 1786089694724403 usec, L: 0
0807 08:01:34.724876 (+ 68us) op_driver.cc:331] REPLICATION: starting
0807 08:01:34.773990 (+ 49114us) log.cc:842] Serialized 3239 byte log entry
0807 08:01:35.283168 (+509178us) op_driver.cc:444] REPLICATION: finished
0807 08:01:35.449164 (+165996us) write_op.cc:301] APPLY: starting
0807 08:01:35.456505 (+ 7341us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:35.481820 (+ 25315us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:35.481957 (+ 137us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:35.582497 (+100540us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:35.582968 (+ 471us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0807 08:01:35.583012 (+ 44us) write_op.cc:312] APPLY: finished
0807 08:01:35.583371 (+ 359us) log.cc:842] Serialized 65 byte log entry
0807 08:01:35.583557 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:35.583690 (+ 133us) write_op.cc:454] Released schema lock
0807 08:01:35.583753 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:35.584191 (+ 438us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:35.584219 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":152188,"apply.run_cpu_time_us":5166,"apply.run_wall_time_us":146603,"num_ops":8,"prepare.queue_time_us":667811,"prepare.run_cpu_time_us":14074,"prepare.run_wall_time_us":399614,"raft.queue_time_us":226293,"raft.run_cpu_time_us":45815,"raft.run_wall_time_us":108628,"replication_time_us":528489,"thread_start_us":825315,"threads_started":4,"wal-append.queue_time_us":123871}]]}
I20260807 08:01:35.928786 17226 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 08:01:35.969748 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 138945us
W20260807 08:01:35.970929 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 199094us
W20260807 08:01:35.970474 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 77999.2us
I20260807 08:01:35.974284 13428 catalog_manager.cc:5495] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:35.883111 16976 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:35.940428 17242 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:35.981472 14583 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.985049 13540 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.968461 17226 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:35.990828 13808 raft_consensus.cc:1261] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 08:01:35.993813 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:35.988334 16893 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:35.996620 14583 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:35.997804 14583 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:35.993817 17226 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:36.012028 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:36.015753 13428 catalog_manager.cc:5495] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 2 to 3, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 3 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } }
I20260807 08:01:36.018844 13428 catalog_manager.cc:5495] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 2 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 3 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:35.993691 17256 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.021s
I20260807 08:01:36.009404 17315 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:36.012032 17116 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:36.023586 17316 consensus_queue.cc:1046] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:36.027305 13428 catalog_manager.cc:5495] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 3 to 4, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 4 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:36.032003 13540 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:36.013720 17303 consensus_queue.cc:1046] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.015s
I20260807 08:01:36.045994 17303 raft_consensus.cc:3032] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:36.046384 17303 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:36.033926 17116 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:36.049396 17190 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:36.039728 16459 raft_consensus.cc:2781] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:36.052548 16459 raft_consensus.cc:686] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:36.055050 16459 consensus_queue.cc:237] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.055637 16976 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:36.020294 17141 consensus_queue.cc:1035] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260807 08:01:36.058177 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:36.020431 13805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:36.058956 13805 raft_consensus.cc:595] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:36.059326 13805 raft_consensus.cc:603] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:36.047360 17303 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.063541 16992 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:36.064049 17303 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:36.070698 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.071475 13808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
mode: GRACEFUL
new_leader_uuid: "77da774f2bb54cf28302d53d0db224ad"
from {username='slave'} at 127.0.0.1:46864
I20260807 08:01:36.072000 14629 raft_consensus.cc:1226] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:36.072113 13808 raft_consensus.cc:595] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Received request to transfer leadership to TS 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:36.072567 13808 raft_consensus.cc:603] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:36.074684 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:36.075553 13584 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 6: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:36.077203 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:36.078075 13580 raft_consensus.cc:2379] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 3: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:36.078382 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:36.079212 13588 raft_consensus.cc:2354] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:36.079563 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260807 08:01:36.080215 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:36.082296 17362 log.cc:924] Time spent T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260807 08:01:36.068894 16816 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:36.083204 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.087785 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:36.097926 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:36.098893 13579 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:36.101588 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:36.102411 13587 raft_consensus.cc:2354] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7.
I20260807 08:01:36.102381 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:36.103828 13428 catalog_manager.cc:5495] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 2 to 3, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 3 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:36.105489 13428 catalog_manager.cc:5495] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 2 to 5, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 5 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } }
I20260807 08:01:36.111557 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:36.112394 13808 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:36.117403 16816 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:36.118499 13702 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:36.118778 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:36.120983 13428 catalog_manager.cc:5495] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:36.089244 16992 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:36.120182 14618 raft_consensus.cc:1226] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:36.126356 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:36.120723 13584 raft_consensus.cc:2354] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:36.130498 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.131479 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.131791 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.132340 13709 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:36.133731 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.134763 14629 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:36.130692 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.135413 13590 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260807 08:01:36.135883 14620 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:36.136801 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:36.138041 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.138377 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:36.139114 14618 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:36.139477 13588 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:36.110031 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:36.017691 17242 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:36.167347 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 145281us
I20260807 08:01:36.169471 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:36.170400 13816 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:36.172030 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:36.178208 13540 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:36.178664 13808 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:36.180414 13580 raft_consensus.cc:1226] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:36.188179 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.191141 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.197216 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:36.198622 13537 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:36.090770 17226 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.167744 17315 raft_consensus.cc:686] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:36.209681 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.213253 13540 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:36.196439 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.209100 17256 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:36.188719 17218 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:36.231223 13584 raft_consensus.cc:1261] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 08:01:36.211356 16992 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:36.211969 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.237716 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.238369 13702 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:36.242156 13588 raft_consensus.cc:1261] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260807 08:01:36.247639 17242 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:36.262784 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.260391 13702 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
W20260807 08:01:36.260401 17373 log.cc:924] Time spent T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20260807 08:01:36.269670 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.272315 13702 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:36.260696 17116 raft_consensus.cc:2726] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:36.286458 13710 raft_consensus.cc:1261] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260807 08:01:36.292924 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.294034 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.232399 17218 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:36.297825 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 73107.2us
I20260807 08:01:36.302445 17355 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260807 08:01:36.309932 13702 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:36.317078 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:36.318924 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 114726us
I20260807 08:01:36.319465 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.322105 17159 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:36.322710 17159 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.324980 17159 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:36.264354 17315 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.327097 13709 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:36.328612 17390 consensus_queue.cc:1046] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:36.330281 16816 raft_consensus.cc:3032] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:36.330786 16816 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:36.330976 17226 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:36.331818 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:36.332458 13804 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:36.331918 16816 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.334527 14622 raft_consensus.cc:2396] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:36.335144 16816 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:36.335366 17218 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:36.342860 13702 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:36.343020 17302 raft_consensus.cc:3032] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:36.352756 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.355392 13709 raft_consensus.cc:1226] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:36.357793 13540 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:36.361454 13807 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:36.366251 13816 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:36.369588 13537 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:36.320058 17256 raft_consensus.cc:686] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:36.347620 17302 raft_consensus.cc:729] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:36.387909 13710 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:36.394176 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.391278 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.390560 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.397390 13709 raft_consensus.cc:2362] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6. Re-sending same reply.
I20260807 08:01:36.406999 13804 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:36.407778 17302 consensus_queue.cc:260] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.408288 17218 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:36.408946 17302 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:36.409041 17218 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.411628 17218 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:36.373915 17196 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:36.385051 17116 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:36.418541 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.419157 13702 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:36.417876 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.421051 13710 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:36.391506 17190 consensus_queue.cc:1046] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260807 08:01:36.422490 17190 raft_consensus.cc:3032] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:36.422832 17190 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:36.423633 17190 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:36.424758 17190 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:36.430709 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.429756 13702 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:36.432653 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.433068 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.433598 14622 raft_consensus.cc:1226] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:36.433832 14618 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:36.434029 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.446353 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.450062 13702 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:36.859517 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.860155 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
W20260807 08:01:36.737403 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 74585.7us
I20260807 08:01:36.866532 14620 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:36.866971 14622 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 3: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:36.569216 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.867763 13661 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:36.868321 13705 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:36.468348 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:36.466501 13415 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet b09437796639470caac9548c71d72fb2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 7a889001d6494fe5b7cbbe7da1dae302 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:36.462388 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:36.873797 13415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 407530us
I20260807 08:01:36.875418 13708 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260807 08:01:36.836527 13769 connection.cc:232] RPC call timeout handler was delayed by -4.6617696662433445s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
W20260807 08:01:36.876941 13769 connection.cc:232] RPC call timeout handler was delayed by -4.5528696661349386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1776s for a response
W20260807 08:01:36.494894 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 83949.1us
I20260807 08:01:36.450193 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.475905 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.870312 17141 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 08:01:36.485795 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 115007us
I20260807 08:01:36.879760 13711 raft_consensus.cc:1261] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:36.880301 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:36.870821 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.872546 13701 raft_consensus.cc:2354] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:36.873104 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.884527 13706 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:36.869336 13661 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:36.877290 13769 connection.cc:232] RPC call timeout handler was delayed by -4.52136966609396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
W20260807 08:01:36.885810 13769 connection.cc:232] RPC call timeout handler was delayed by -4.4169696662575006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20260807 08:01:36.886081 13769 connection.cc:232] RPC call timeout handler was delayed by -4.4043696662411094s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
I20260807 08:01:36.887923 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.888542 13702 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:36.890060 13700 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:36.893875 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.881771 14618 raft_consensus.cc:1226] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260807 08:01:36.487361 17116 raft_consensus.cc:686] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:36.898648 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.903347 14627 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:36.904160 13808 raft_consensus.cc:1261] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 08:01:36.894263 17141 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:36.915207 16791 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:36.905022 17390 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:36.918646 16791 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.921365 17390 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:36.922890 16791 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:36.923413 17390 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:36.486555 17256 consensus_queue.cc:237] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:36.926434 13710 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:36.908619 17367 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:36.900087 17218 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:36.927418 17367 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.928236 17218 raft_consensus.cc:686] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:36.929229 13710 raft_consensus.cc:1226] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:36.929174 17218 consensus_queue.cc:237] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.932372 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.933168 13710 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:36.933547 17367 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:36.921289 17116 consensus_queue.cc:237] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:36.937299 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:36.938117 13705 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 3: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:36.938820 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.906368 16459 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:36.941258 16459 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:36.946604 14618 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:36.946893 16459 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:36.949273 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.951511 13701 raft_consensus.cc:1261] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:36.954761 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.956750 13703 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:36.966585 13710 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:36.966578 14622 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:36.974913 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.975524 14622 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:36.976703 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 107480us
I20260807 08:01:36.978646 13591 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:36.985972 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.987054 14629 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:36.985317 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:36.989636 14622 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:36.989558 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:36.990599 14626 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:36.991506 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:36.992664 13700 raft_consensus.cc:2379] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 3: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:37.001195 13706 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:37.002969 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:37.004176 13710 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:37.024843 17190 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:37.025550 17190 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:37.026083 16816 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:37.026597 16816 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:37.028343 17190 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:37.028618 16816 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:37.023582 16992 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:37.028795 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.029292 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:37.029462 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.029250 16992 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:37.030187 13710 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:37.031134 13590 raft_consensus.cc:1261] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260807 08:01:37.031515 13590 pending_rounds.cc:76] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad: Aborting all ops after (but not including) 1
I20260807 08:01:37.030360 13584 raft_consensus.cc:2354] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:37.030239 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:37.031749 13590 pending_rounds.cc:97] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260807 08:01:37.034963 13590 raft_consensus.cc:2866] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:36.986974 17141 consensus_queue.cc:1046] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
W20260807 08:01:37.036582 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 141038us
I20260807 08:01:37.037559 13807 raft_consensus.cc:3032] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:37.042045 13807 raft_consensus.cc:729] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:37.020125 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:37.045269 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.045131 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.044833 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.023818 17256 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:37.048395 14627 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
W20260807 08:01:37.058193 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 103942us
I20260807 08:01:37.054672 13807 consensus_queue.cc:260] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:37.067502 13807 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:37.057349 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:37.068228 14583 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:37.077216 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.087356 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.090099 13585 raft_consensus.cc:2354] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3.
I20260807 08:01:37.092149 13701 raft_consensus.cc:2379] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:37.086902 16893 consensus_queue.cc:1046] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260807 08:01:37.094048 16893 raft_consensus.cc:3032] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:37.094412 16893 raft_consensus.cc:729] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:37.095258 16893 consensus_queue.cc:260] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:37.096526 17367 raft_consensus.cc:3032] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:37.097127 17367 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:37.066774 16992 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:37.117918 13661 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:37.105705 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 47741.3us
W20260807 08:01:37.145876 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 77717.7us
I20260807 08:01:37.098843 13590 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:37.165968 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:37.170825 13710 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:37.174006 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 119988us
I20260807 08:01:37.056275 17190 consensus_queue.cc:1046] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.108s
I20260807 08:01:37.179992 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.086983 17397 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:37.187608 13661 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:37.097731 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:37.179992 17141 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:37.216676 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.224696 13769 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:37.192764 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:37.173530 16893 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:37.242102 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.087049 17256 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:37.497483 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.502128 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:37.098392 17367 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:37.243551 17141 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:37.424616 13706 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:37.235105 13701 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:37.212692 17368 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:37.239539 13661 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:37.191154 16976 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 08:01:37.469156 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 256564us
I20260807 08:01:37.247090 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:37.479492 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:37.566992 13703 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:37.467100 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:37.492401 13710 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:37.572325 13808 raft_consensus.cc:2396] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:37.623890 13585 raft_consensus.cc:1226] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260807 08:01:37.656468 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 456889us
I20260807 08:01:37.670958 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:37.562036 17401 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.112s
I20260807 08:01:37.680264 13591 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260807 08:01:37.697343 13590 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:37.706373 13592 raft_consensus.cc:2396] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 08:01:37.577837 17367 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:37.722183 13769 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:37.566850 16976 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:37.742069 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:37.779237 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:37.799853 13808 raft_consensus.cc:2396] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:37.776419 13585 raft_consensus.cc:1226] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260807 08:01:37.819090 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:37.766101 14583 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 2.186s (SENT) [suppressed 14 similar messages]
W20260807 08:01:37.864262 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 102843us
I20260807 08:01:37.870855 13805 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
W20260807 08:01:37.892949 14583 leader_election.cc:334] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.521s (SENT)
I20260807 08:01:37.753014 16976 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:37.899852 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:37.904946 13706 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:37.910955 13587 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:37.746686 17387 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:37.923394 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 201958us
I20260807 08:01:37.844754 17190 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:37.941051 13809 raft_consensus.cc:1261] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 08:01:37.842826 17256 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:37.943235 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:37.946547 13811 raft_consensus.cc:1261] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260807 08:01:37.946710 17420 log.cc:924] Time spent T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.200s user 0.000s sys 0.001s
I20260807 08:01:37.952416 13808 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 08:01:37.913162 17431 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:37.952659 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 249412us
W20260807 08:01:37.957502 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 222923us
I20260807 08:01:37.969223 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:37.969614 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 730547us
I20260807 08:01:37.972424 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:37.977188 13592 raft_consensus.cc:2379] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:37.934816 17218 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260807 08:01:37.985533 13591 raft_consensus.cc:1226] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:37.944448 17427 raft_consensus.cc:2741] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 08:01:37.990821 17141 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:37.990834 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:37.971319 17401 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:37.942467 17429 consensus_queue.cc:1046] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20260807 08:01:38.004050 14626 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:38.004050 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:38.004626 13805 raft_consensus.cc:1261] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260807 08:01:38.005378 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:38.006055 13591 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:37.955955 17387 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.009402 13587 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:38.000077 17421 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:38.014058 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:38.004667 17427 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:38.019716 17429 raft_consensus.cc:3032] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:38.022856 17429 raft_consensus.cc:729] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:38.024070 14623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 517) took 1054 ms. Trace:
I20260807 08:01:38.023737 17429 consensus_queue.cc:260] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.025063 17429 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:38.026736 13807 consensus_meta.cc:287] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.959s user 0.022s sys 0.006s
I20260807 08:01:38.019747 13587 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:38.028616 13807 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260807 08:01:38.028962 13807 pending_rounds.cc:76] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 4
I20260807 08:01:38.029208 13807 pending_rounds.cc:97] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260807 08:01:38.029124 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.029611 13807 raft_consensus.cc:2866] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:38.030352 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 483) took 1008 ms. Trace:
W20260807 08:01:38.014576 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 206636us
I20260807 08:01:38.025336 13701 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:38.031075 13807 rpcz_store.cc:276] 0807 08:01:37.022891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:37.032297 (+ 9406us) service_pool.cc:227] Handling call
0807 08:01:37.037396 (+ 5099us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:38.030074 (+992678us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:38.030288 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:38.032188 13540 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:38.032716 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 79623us
W20260807 08:01:38.032923 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 56101.8us
I20260807 08:01:38.004566 17440 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 08:01:38.034117 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 141393us
I20260807 08:01:38.041249 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.020962 16976 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.044970 17442 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.045107 17390 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:38.045394 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:37.973560 17431 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.057740 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:38.058499 14627 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:38.033265 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.059837 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:38.060446 14626 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:38.061924 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.062219 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:38.063318 16893 consensus_meta.cc:287] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.821s user 0.030s sys 0.000s
I20260807 08:01:38.061223 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.068874 13585 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:38.004911 17401 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:38.066517 14583 leader_election.cc:334] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.629s (SENT)
W20260807 08:01:38.078023 14583 leader_election.cc:334] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.738s (SENT)
I20260807 08:01:38.079237 17368 raft_consensus.cc:3032] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:38.079665 17368 raft_consensus.cc:729] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:38.080507 17368 consensus_queue.cc:260] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.081758 17368 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:38.055047 17421 raft_consensus.cc:2741] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260807 08:01:38.024500 14623 rpcz_store.cc:276] 0807 08:01:36.969808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:36.972701 (+ 2893us) service_pool.cc:227] Handling call
0807 08:01:36.990515 (+ 17814us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:36.991085 (+ 570us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:36.991127 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:36.991894 (+ 767us) log.cc:842] Serialized 52 byte log entry
0807 08:01:37.089674 (+ 97780us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:37.288620 (+198946us) log.cc:842] Serialized 14 byte log entry
0807 08:01:37.588779 (+300159us) log.cc:842] Serialized 14 byte log entry
0807 08:01:37.761061 (+172282us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:37.818082 (+ 57021us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:38.023808 (+205726us) raft_consensus.cc:1671] finished
0807 08:01:38.023837 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:38.023990 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50123,"prepare.queue_time_us":593443,"prepare.run_cpu_time_us":9546,"prepare.run_wall_time_us":246315,"raft.queue_time_us":135426,"raft.run_cpu_time_us":1130,"raft.run_wall_time_us":49156,"spinlock_wait_cycles":27700480,"thread_start_us":86043,"threads_started":1,"wal-append.queue_time_us":957636}
I20260807 08:01:38.086100 17401 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:38.086515 13769 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.087989 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.089190 17190 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:38.089661 17421 raft_consensus.cc:2741] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260807 08:01:38.089715 17190 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.090416 17421 consensus_queue.cc:1035] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260807 08:01:38.091817 17190 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:38.091969 17421 raft_consensus.cc:2741] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260807 08:01:38.091984 17446 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:38.098680 17431 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:38.099965 17387 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:38.027669 17427 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.102138 17427 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:38.102789 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 70692.4us
I20260807 08:01:38.104241 13540 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:38.105640 13700 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:38.105686 17387 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:38.111364 17387 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:38.111930 17387 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:38.045131 16976 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:38.078687 14583 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.118494 16893 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.121102 13591 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:38.122732 16893 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:38.141765 17218 consensus_queue.cc:1035] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:38.124272 17447 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260807 08:01:38.143247 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 91826.1us
I20260807 08:01:38.089414 13769 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.144232 17218 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260807 08:01:38.144755 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 55402.9us
I20260807 08:01:38.145354 13592 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:38.126549 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 141917us
I20260807 08:01:38.147419 16893 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:38.147603 17218 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:38.132809 14583 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:38.134907 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.150394 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.150965 17423 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:38.151350 17397 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:38.151571 17423 raft_consensus.cc:686] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:38.151971 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.152104 17397 consensus_queue.cc:1035] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:01:38.152557 17423 consensus_queue.cc:237] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.135361 16976 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:38.155745 16976 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.157944 17303 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.158569 16976 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.136224 17190 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:38.166306 17190 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.168449 17190 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.170083 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.136363 17450 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:38.136749 17367 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:38.137833 13807 raft_consensus.cc:2396] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260807 08:01:38.140168 17387 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.172232 17367 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.173696 17387 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.145649 13769 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.174223 17367 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.105441 13540 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.145900 17368 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:38.174825 17450 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:38.175365 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 69992.7us
I20260807 08:01:38.175099 17368 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.145988 17429 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:38.176180 13540 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.176265 17429 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:38.176782 17429 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:38.176903 17450 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:38.135674 17443 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:38.177414 17368 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.177829 17443 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.179129 17401 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:38.179527 17401 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:38.179895 17443 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.179930 17401 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:38.180642 17442 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.181218 17442 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:38.182926 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.184015 13807 raft_consensus.cc:2396] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260807 08:01:38.160506 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.135864 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:38.185807 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 50018.6us
I20260807 08:01:38.187517 13661 leader_election.cc:398] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:38.187815 13661 leader_election.cc:401] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:38.188851 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:38.161710 13659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.014s sys 0.009s
W20260807 08:01:38.189702 13659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.668s user 0.014s sys 0.009s
W20260807 08:01:38.190163 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 44647.2us
I20260807 08:01:38.190086 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.190490 17367 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:38.191205 17432 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:38.192332 17432 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:38.164357 17218 consensus_queue.cc:1035] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:38.192873 13812 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:38.192775 17432 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.194339 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.194847 17432 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.194967 13814 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:38.194878 17426 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:38.195453 17367 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:38.195672 17426 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.197058 13657 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.197930 17190 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:38.198177 17426 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.198318 17190 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:38.198949 17190 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:38.200214 13661 leader_election.cc:398] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7.
I20260807 08:01:38.200618 13661 leader_election.cc:401] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260807 08:01:38.201707 13661 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:38.202782 17368 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7.
I20260807 08:01:38.203657 17368 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.206375 13657 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.211160 17450 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:38.211539 17450 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:38.212049 17450 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:01:38.212769 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 47692.8us
I20260807 08:01:38.214218 13661 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:38.215354 17429 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.216724 13804 raft_consensus.cc:1226] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:38.217971 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.219069 13814 raft_consensus.cc:2396] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260807 08:01:38.220316 17427 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:38.221740 17368 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260807 08:01:38.223279 13769 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.224167 17427 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:38.224565 17427 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:38.225231 17427 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:38.224859 13430 catalog_manager.cc:5495] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:38.226985 13430 catalog_manager.cc:5495] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 3 to 5, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 5 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:38.228657 13430 catalog_manager.cc:5495] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 2 to 6, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 6 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:38.177870 13540 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.229780 17368 consensus_queue.cc:1046] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:38.230715 17387 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:38.231166 16976 raft_consensus.cc:3032] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:38.230368 13430 catalog_manager.cc:5495] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 4, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:38.231616 16976 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:38.233188 13769 leader_election.cc:398] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:38.232646 16976 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.235477 16976 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:38.236907 16791 consensus_queue.cc:1046] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:38.237119 13430 catalog_manager.cc:5495] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 3 to 5, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 5 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260807 08:01:38.230446 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 52653.4us
I20260807 08:01:38.231618 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.238227 16791 raft_consensus.cc:3032] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:38.241492 16791 raft_consensus.cc:729] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:38.242457 16791 consensus_queue.cc:260] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.238937 13661 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:38.239997 13812 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:38.233500 13769 leader_election.cc:401] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:38.240316 13430 catalog_manager.cc:5495] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 3 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:38.240602 13540 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.246572 13430 catalog_manager.cc:5495] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:38.245318 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:38.243971 13428 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56596 (request call id 308) took 11003 ms. Trace:
I20260807 08:01:38.249104 13428 rpcz_store.cc:276] 0807 08:01:27.240692 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:27.241699 (+ 1007us) service_pool.cc:227] Handling call
0807 08:01:38.243890 (+11002191us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:36.855569 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:36.856427 (+ 858us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:36.856505 (+ 78us) write_op.cc:435] Acquired schema lock
0807 08:01:36.856539 (+ 34us) tablet.cc:660] Decoding operations
0807 08:01:36.899374 (+ 42835us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:36.909418 (+ 10044us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:36.914318 (+ 4900us) tablet.cc:683] Acquiring locks for 11 operations
0807 08:01:37.078246 (+163928us) tablet.cc:699] Row locks acquired
0807 08:01:37.086078 (+ 7832us) write_op.cc:260] PREPARE: finished
0807 08:01:37.157460 (+ 71382us) write_op.cc:270] Start()
0807 08:01:37.169920 (+ 12460us) write_op.cc:276] Timestamp: P: 1786089697135049 usec, L: 0
0807 08:01:37.176277 (+ 6357us) op_driver.cc:331] REPLICATION: starting
0807 08:01:37.666378 (+490101us) log.cc:842] Serialized 4412 byte log entry
0807 08:01:38.067709 (+401331us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b180022efa0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:38.131133 (+ 63424us) op_driver.cc:444] REPLICATION: finished
0807 08:01:38.184578 (+ 53445us) write_op.cc:301] APPLY: starting
0807 08:01:38.184656 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:38.184890 (+ 234us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:38.184911 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:38.186233 (+ 1322us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:38.186483 (+ 250us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0807 08:01:38.186538 (+ 55us) write_op.cc:312] APPLY: finished
0807 08:01:38.186815 (+ 277us) log.cc:842] Serialized 83 byte log entry
0807 08:01:38.190066 (+ 3251us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:38.190211 (+ 145us) write_op.cc:454] Released schema lock
0807 08:01:38.190259 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:38.204195 (+ 13936us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:38.204225 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":36060,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":15441,"apply.run_cpu_time_us":3377,"apply.run_wall_time_us":20074,"num_ops":11,"prepare.queue_time_us":589530,"prepare.run_cpu_time_us":101732,"prepare.run_wall_time_us":1123588,"raft.queue_time_us":50089,"raft.run_cpu_time_us":14069,"raft.run_wall_time_us":53306,"replication_time_us":948000,"spinlock_wait_cycles":4878976,"thread_start_us":498086,"threads_started":4,"wal-append.queue_time_us":321561}]]}
I20260807 08:01:38.248588 13430 catalog_manager.cc:5495] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 1 to 4, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:38.257130 13769 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.254506 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:38.220752 17429 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:38.257467 13430 catalog_manager.cc:5495] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 3 to 4, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 4 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:38.260484 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:38.261534 13657 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.264362 13661 leader_election.cc:398] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:38.264706 13661 leader_election.cc:401] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:38.268138 13540 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.259234 16791 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:38.275099 17426 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:38.275585 17426 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:38.275985 17426 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:38.275144 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:38.277693 17141 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:01:38.272480 17421 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:38.258361 17368 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:38.285763 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:38.286454 13661 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:38.286424 13706 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:38.293476 17256 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.296286 13701 raft_consensus.cc:1226] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 08:01:38.297847 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:38.298584 13700 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:38.299626 13591 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:38.309422 13591 raft_consensus.cc:595] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:38.309855 13591 raft_consensus.cc:603] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:38.318431 17427 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.236913 17401 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.300796 13540 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:38.307078 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:38.322314 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:38.323055 14623 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:38.267380 17443 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.324249 17438 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:38.265404 17387 raft_consensus.cc:2726] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:38.332499 17367 consensus_queue.cc:1035] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:38.333137 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:38.324971 17438 consensus_queue.cc:1035] T 6f5ccbaa50ae4451a35038284693254c P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:38.336222 13706 raft_consensus.cc:1226] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260807 08:01:38.319265 17450 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.257851 17190 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.309043 17368 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.307840 17421 raft_consensus.cc:686] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:38.347663 13540 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:38.350337 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:38.358752 13661 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:38.366619 16992 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:38.343173 17218 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:38.375593 13587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:38.376214 13587 raft_consensus.cc:595] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.376549 13587 raft_consensus.cc:603] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:38.341043 17446 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:38.375702 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:38.382019 13706 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:38.374931 13769 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:38.380120 13661 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:38.328011 17256 consensus_queue.cc:1035] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.020s
I20260807 08:01:38.327821 17442 consensus_queue.cc:1035] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260807 08:01:38.401885 13661 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:38.404091 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:38.404965 13705 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
W20260807 08:01:38.439255 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 64395.5us
I20260807 08:01:38.448274 17190 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.449256 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:38.449638 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:38.449870 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:38.450217 14627 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:38.450482 13705 raft_consensus.cc:2362] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7. Re-sending same reply.
I20260807 08:01:38.451083 17431 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:38.451376 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:38.451640 17431 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.454161 14622 raft_consensus.cc:1226] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:38.461619 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.464568 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.466075 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:38.467293 13706 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:38.467120 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.464699 13587 raft_consensus.cc:2396] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 08:01:38.469882 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.470697 13584 raft_consensus.cc:2396] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:01:38.472002 13708 raft_consensus.cc:1226] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:38.473608 13769 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:38.473971 13709 raft_consensus.cc:1226] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 2. Current term is 3. Ops: []
I20260807 08:01:38.476878 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:38.504760 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 132277us
I20260807 08:01:38.464560 14627 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:38.455621 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:38.437399 17367 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.512801 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.440935 17141 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:38.438570 17432 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:38.513074 13540 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:38.457062 17429 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:38.519268 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:38.402426 17440 consensus_queue.cc:1046] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.118s
I20260807 08:01:38.514182 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.613313 13585 raft_consensus.cc:2379] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
W20260807 08:01:38.521010 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 47545.8us
I20260807 08:01:38.520172 14620 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:38.588661 17401 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:38.588542 17431 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:38.588725 16992 raft_consensus.cc:686] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:38.630185 13769 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:38.600238 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.598443 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.597075 13706 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:38.601267 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:38.632067 13709 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:38.589494 17368 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:38.595381 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:38.635226 13587 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:38.606236 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:38.597563 17242 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:38.597996 17429 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:38.621673 17256 consensus_queue.cc:1046] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:38.625116 13705 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:38.635749 13661 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:38.636235 13708 raft_consensus.cc:2362] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6. Re-sending same reply.
I20260807 08:01:38.640067 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:38.602938 17141 raft_consensus.cc:686] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:38.593392 17439 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260807 08:01:38.657474 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 182566us
I20260807 08:01:38.663380 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:38.663928 13705 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:38.666007 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:38.666813 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:38.684751 13709 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:38.686441 13709 raft_consensus.cc:1226] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:38.643522 17218 raft_consensus.cc:686] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:38.772717 17218 consensus_queue.cc:237] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.760916 17141 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.644744 17427 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:38.776153 17427 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:38.653337 17429 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:38.662346 14618 raft_consensus.cc:2379] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 6: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:38.695734 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:38.793210 13708 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:38.656704 17431 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 08:01:38.690850 13415 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet b09437796639470caac9548c71d72fb2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 7a889001d6494fe5b7cbbe7da1dae302 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:38.667102 17423 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:38.720263 17457 log.cc:924] Time spent T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20260807 08:01:38.668159 17450 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:38.723706 17477 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260807 08:01:38.726047 17468 log.cc:924] Time spent T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
W20260807 08:01:38.732137 17473 log.cc:924] Time spent T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260807 08:01:38.521673 17421 consensus_queue.cc:237] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:38.737207 17460 log.cc:924] Time spent T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.273s user 0.000s sys 0.001s
I20260807 08:01:38.738076 17426 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.678275 17401 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:38.644366 17368 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.747314 16992 consensus_queue.cc:237] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:38.671391 13700 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260807 08:01:38.759711 16976 consensus_meta.cc:287] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.524s user 0.008s sys 0.001s
I20260807 08:01:38.760195 17432 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:38.653597 17242 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:38.662608 17439 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:38.729279 17442 raft_consensus.cc:3032] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:38.794613 13705 raft_consensus.cc:1226] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 3. Current term is 4. Ops: []
W20260807 08:01:38.806713 13415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 115922us
I20260807 08:01:38.808580 13590 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:38.817111 13432 catalog_manager.cc:5495] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 4 to 5, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 5 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:38.776536 17427 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:38.964105 13703 raft_consensus.cc:2354] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:38.876103 13430 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57868 (request call id 215) took 3115 ms. Trace:
I20260807 08:01:38.893678 13708 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260807 08:01:38.900538 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 387561us
W20260807 08:01:38.901146 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 226634us
I20260807 08:01:38.809223 17367 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:38.938210 13591 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.001s sys 0.019s
W20260807 08:01:38.968480 13591 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.635s user 0.002s sys 0.019s
I20260807 08:01:38.809877 17431 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:38.945700 13592 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.014s sys 0.008s
I20260807 08:01:38.810447 17449 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:38.957077 13710 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:38.955147 13432 catalog_manager.cc:5495] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 2 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 3 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:38.890635 16893 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:38.976032 13592 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.567s user 0.015s sys 0.008s
I20260807 08:01:38.952479 17442 raft_consensus.cc:729] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:39.114996 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 442528us
I20260807 08:01:39.115326 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:39.115655 13432 catalog_manager.cc:5495] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } }
I20260807 08:01:38.957907 17439 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:39.117818 13432 catalog_manager.cc:5495] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 2 to 4, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 4 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } }
I20260807 08:01:38.952108 17443 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:38.974694 17190 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:39.138921 13432 catalog_manager.cc:5495] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 6 to 7, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 7 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } }
I20260807 08:01:39.144716 13590 raft_consensus.cc:595] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.116001 13708 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:38.975162 17427 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:38.955300 17401 raft_consensus.cc:686] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:38.975714 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:38.876794 17423 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.180233 16791 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:38.998142 17431 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:39.239552 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:39.452711 17490 log.cc:924] Time spent T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
I20260807 08:01:38.984357 17367 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:39.262141 13432 catalog_manager.cc:5495] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 3, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:39.182749 17443 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:39.109929 16893 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.069059 17449 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.258697 17190 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:39.234714 17461 log.cc:924] Time spent T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
W20260807 08:01:39.306715 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 670994us
I20260807 08:01:39.470598 13661 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:39.313416 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 52860.9us
I20260807 08:01:39.228924 14618 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:39.234340 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:39.237790 16791 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:39.394584 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 155789us
I20260807 08:01:39.324827 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:39.342288 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 226521us
I20260807 08:01:39.477027 13540 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:39.455102 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 129649us
I20260807 08:01:38.964795 13430 rpcz_store.cc:276] 0807 08:01:35.701780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:35.702139 (+ 359us) service_pool.cc:227] Handling call
0807 08:01:38.816507 (+3114368us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:38.294573 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:38.295097 (+ 524us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:38.295151 (+ 54us) write_op.cc:435] Acquired schema lock
0807 08:01:38.295171 (+ 20us) tablet.cc:660] Decoding operations
0807 08:01:38.296141 (+ 970us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:38.296198 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:38.296232 (+ 34us) tablet.cc:683] Acquiring locks for 9 operations
0807 08:01:38.297023 (+ 791us) tablet.cc:699] Row locks acquired
0807 08:01:38.297043 (+ 20us) write_op.cc:260] PREPARE: finished
0807 08:01:38.297188 (+ 145us) write_op.cc:270] Start()
0807 08:01:38.297326 (+ 138us) write_op.cc:276] Timestamp: P: 1786089698297140 usec, L: 0
0807 08:01:38.297349 (+ 23us) op_driver.cc:331] REPLICATION: starting
0807 08:01:38.351677 (+ 54328us) log.cc:842] Serialized 3597 byte log entry
0807 08:01:38.453922 (+102245us) op_driver.cc:444] REPLICATION: finished
0807 08:01:38.454535 (+ 613us) write_op.cc:301] APPLY: starting
0807 08:01:38.454612 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:38.454849 (+ 237us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:38.454943 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:38.480899 (+ 25956us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:38.481091 (+ 192us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0807 08:01:38.481135 (+ 44us) write_op.cc:312] APPLY: finished
0807 08:01:38.481412 (+ 277us) log.cc:842] Serialized 71 byte log entry
0807 08:01:38.512815 (+ 31403us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:38.600847 (+ 88032us) write_op.cc:454] Released schema lock
0807 08:01:38.614847 (+ 14000us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:38.659957 (+ 45110us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:38.662361 (+ 2404us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6823,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":380,"apply.run_cpu_time_us":15772,"apply.run_wall_time_us":239172,"num_ops":9,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":4011,"prepare.run_wall_time_us":83204,"replication_time_us":156476}]]}
I20260807 08:01:39.478221 13809 raft_consensus.cc:3032] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:39.478686 13809 raft_consensus.cc:729] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:01:39.460887 13711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.002s sys 0.001s
I20260807 08:01:39.213999 14626 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:39.463531 13432 catalog_manager.cc:5495] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 4, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 4 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
W20260807 08:01:39.479457 13711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s user 0.002s sys 0.001s
W20260807 08:01:39.455423 17141 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.010s sys 0.041s
I20260807 08:01:39.480600 13432 catalog_manager.cc:5495] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 4, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 4 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:39.483747 17401 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.484606 17367 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.486919 17478 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:39.487337 17478 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:39.487287 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:39.487917 17478 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:39.481884 13432 catalog_manager.cc:5495] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:39.490599 16791 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:39.491791 13432 catalog_manager.cc:5495] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 3 to 4, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 4 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:39.493626 13432 catalog_manager.cc:5495] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 1 to 4, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 4 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:39.495373 13432 catalog_manager.cc:5495] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 reported cstate change: term changed from 2 to 5, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67). New cstate: current_term: 5 leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:39.494274 17256 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:39.497287 17256 raft_consensus.cc:686] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:39.497653 17141 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s user 0.010s sys 0.041s
I20260807 08:01:39.461478 17303 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:39.498283 17256 consensus_queue.cc:237] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.498755 17303 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:39.522107 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 44583.4us
I20260807 08:01:39.264415 17442 consensus_queue.cc:260] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:39.516011 13702 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.012s sys 0.009s
W20260807 08:01:39.561681 13702 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.606s user 0.013s sys 0.009s
I20260807 08:01:39.519807 13661 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:39.565791 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 46088.6us
I20260807 08:01:39.563836 13809 consensus_queue.cc:260] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.499619 17447 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:39.578285 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:39.579087 13807 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 6
I20260807 08:01:39.580462 13809 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:39.585201 13661 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:39.586575 13769 leader_election.cc:334] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 2.075s (SENT)
I20260807 08:01:39.598704 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:39.606918 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:39.604235 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:39.608145 13588 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:39.604542 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:39.605775 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 134192us
I20260807 08:01:39.581810 17368 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.559300 17483 consensus_queue.cc:1035] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.098s
I20260807 08:01:39.620920 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:39.582598 17242 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.623023 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 367) took 1203 ms. Trace:
I20260807 08:01:39.582479 17426 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.625365 17515 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
W20260807 08:01:39.626482 17500 log.cc:924] Time spent T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260807 08:01:39.585120 17447 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.606511 17442 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:39.626900 17390 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:39.606086 17491 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:39.633162 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1738) took 1327 ms. Trace:
I20260807 08:01:39.609647 17440 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:39.633883 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:39.634590 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 375) took 1227 ms. Trace:
W20260807 08:01:39.634866 17429 consensus_meta.cc:287] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.825s user 0.010s sys 0.000s
I20260807 08:01:39.665876 13709 rpcz_store.cc:276] 0807 08:01:38.406831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.807968 (+401137us) service_pool.cc:227] Handling call
0807 08:01:38.890082 (+ 82114us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:38.974296 (+ 84214us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:38.974329 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:38.974382 (+ 53us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:38.974432 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:39.634361 (+659929us) spinlock_profiling.cc:243] Waited 66.5 ms on lock 0x7b54003a9dd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:39.634412 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:39.634535 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152841472}
I20260807 08:01:39.640457 13809 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 08:01:39.598755 17489 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:39.644302 13708 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:39.604458 17486 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:39.655591 17449 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.641600 13711 rpcz_store.cc:276] 0807 08:01:38.404037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.405981 (+ 1944us) service_pool.cc:227] Handling call
0807 08:01:38.473407 (+ 67426us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:38.475759 (+ 2352us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 08:01:38.957193 (+481434us) log.cc:842] Serialized 14 byte log entry
0807 08:01:39.606973 (+649780us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:39.607025 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:39.607079 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:39.607153 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:39.607273 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10428,"prepare.queue_time_us":986017,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":11609,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":1075625,"threads_started":3,"wal-append.queue_time_us":666466}
W20260807 08:01:39.660614 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 180760us
I20260807 08:01:39.666635 13703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1740) took 1336 ms. Trace:
I20260807 08:01:39.690698 13703 rpcz_store.cc:276] 0807 08:01:38.330280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.330465 (+ 185us) service_pool.cc:227] Handling call
0807 08:01:39.666456 (+1335991us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b5400408154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:39.666581 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1449010560}
I20260807 08:01:39.691139 13540 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:39.667137 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:39.669834 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:39.674676 13661 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.681845 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:39.682538 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 366) took 1278 ms. Trace:
I20260807 08:01:39.692876 13591 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:39.665815 17401 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:39.683107 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:39.683463 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:39.670337 13809 pending_rounds.cc:76] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 5
W20260807 08:01:39.685953 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 162895us
I20260807 08:01:39.656345 17438 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:39.640079 13807 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:39.689218 13705 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260807 08:01:39.683049 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 102732us
I20260807 08:01:39.674001 17190 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:39.692785 13702 rpcz_store.cc:276] 0807 08:01:38.403705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.403829 (+ 124us) service_pool.cc:227] Handling call
0807 08:01:38.404191 (+ 362us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:38.511476 (+107285us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:38.511507 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:38.521192 (+ 9685us) log.cc:842] Serialized 26 byte log entry
0807 08:01:38.901578 (+380386us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:39.562054 (+660476us) log.cc:842] Serialized 14 byte log entry
0807 08:01:39.562949 (+ 895us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:39.563031 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:39.682336 (+119305us) raft_consensus.cc:1671] finished
0807 08:01:39.682354 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:39.682482 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":637,"prepare.queue_time_us":586782,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":51558,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":7929,"spinlock_wait_cycles":39040,"thread_start_us":960086,"threads_started":3,"wal-append.queue_time_us":850556}
I20260807 08:01:39.673193 17423 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:39.672957 17429 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.668097 17390 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:39.674358 17439 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.698643 13540 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:39.778836 13809 pending_rounds.cc:97] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260807 08:01:39.779748 13809 raft_consensus.cc:2866] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 08:01:39.780879 13769 leader_election.cc:334] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 2.137s (SENT)
I20260807 08:01:39.749943 13811 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:39.781497 13769 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:39.779507 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 80982.2us
I20260807 08:01:39.700500 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:39.783038 13540 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:39.688894 16992 raft_consensus.cc:979] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
W20260807 08:01:39.785073 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 84685us
I20260807 08:01:39.787850 13661 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:39.786067 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 102731us
I20260807 08:01:39.673430 17440 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.679489 17515 raft_consensus.cc:686] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:39.752239 17511 log.cc:924] Time spent T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260807 08:01:39.786664 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:39.794426 13661 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.795074 13816 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:39.795471 13814 raft_consensus.cc:1226] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:39.797506 13661 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.797758 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:39.685880 17486 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:39.754418 17533 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:39.797914 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:39.736972 17443 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.802948 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:39.789799 17190 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:39.751166 17447 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:39.722345 16893 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.788995 17477 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:39.795100 17426 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:39.799026 17368 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:39.790546 17449 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:39.791549 17524 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:39.814982 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:39.833844 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:39.805779 16992 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 08:01:39.806371 17446 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260807 08:01:39.672857 17491 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.812513 17478 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.849440 17515 consensus_queue.cc:237] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.850117 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:39.842235 16992 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:39.851320 16992 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:39.852372 17524 raft_consensus.cc:686] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:39.834448 17431 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:39.854852 17427 consensus_meta.cc:287] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.579s user 0.006s sys 0.002s
I20260807 08:01:39.855436 17431 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:39.855284 17427 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.856006 17431 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:39.856477 17431 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:39.753940 17303 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.807366 17486 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:39.796093 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:39.816685 13702 raft_consensus.cc:1261] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 08:01:39.837770 14583 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.806119 17367 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:39.839121 14627 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 08:01:39.850793 17478 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:39.857663 17540 raft_consensus.cc:1067] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:01:39.857287 17427 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:39.722851 17401 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:39.821206 17477 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:39.875432 13711 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260807 08:01:39.875661 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 94816.7us
I20260807 08:01:39.882161 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:39.827373 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:39.886117 13809 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:39.882189 13804 raft_consensus.cc:2379] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:39.788774 17438 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:39.883744 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 46090.4us
I20260807 08:01:39.888871 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:39.831362 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:39.862628 17483 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:39.693953 17489 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:39.893661 13769 leader_election.cc:334] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 2.205s (SENT)
I20260807 08:01:39.895152 14583 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:39.897785 17440 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:39.898664 17401 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:39.863528 17449 raft_consensus.cc:686] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:39.862950 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:39.871861 17367 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:39.834268 17423 consensus_queue.cc:1046] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260807 08:01:39.853370 17524 consensus_queue.cc:237] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.860054 17242 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
I20260807 08:01:39.905860 13769 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:39.904361 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 105631us
I20260807 08:01:39.907688 14583 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.886981 17477 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:39.899300 16459 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:39.911413 13709 raft_consensus.cc:1226] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 08:01:39.891425 17218 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:39.900902 16976 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:39.902299 17303 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:39.916893 13661 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.909431 17447 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:39.923365 13769 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:39.913791 17367 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:39.925714 17550 raft_consensus.cc:2726] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:39.929589 16893 consensus_queue.cc:1046] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:39.930233 16791 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 08:01:39.931064 17439 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:39.924366 17397 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:39.931348 13709 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 08:01:39.939916 13661 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.954967 13769 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:39.955427 13661 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.956120 13769 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:39.956861 13661 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:39.957731 17478 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:39.958169 17440 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:39.958595 17440 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:39.958999 17440 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:39.959120 17478 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:39.960872 17538 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:39.961664 17397 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:39.961410 17538 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.962081 17397 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:39.963716 17538 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:39.972046 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:39.920248 17438 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.934046 17303 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:39.941829 17390 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:39.966941 17421 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 08:01:39.989615 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 177172us
I20260807 08:01:39.665000 13701 rpcz_store.cc:276] 0807 08:01:38.306018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.309327 (+ 3309us) service_pool.cc:227] Handling call
0807 08:01:38.347870 (+ 38543us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:38.765509 (+417639us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b18003f6ca0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a025345 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a040155 00007f6b1cf633d0 00007f6b1cf63f6b 00007f6b1cf6e78b 00007f6b1cf6aedf
0807 08:01:39.138782 (+373273us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:39.146753 (+ 7971us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:39.325431 (+178678us) log.cc:842] Serialized 26 byte log entry
0807 08:01:39.465699 (+140268us) raft_consensus.cc:1631] Marking committed up to 1
0807 08:01:39.465742 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:39.465817 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:39.632796 (+166979us) raft_consensus.cc:1671] finished
0807 08:01:39.632825 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:39.633089 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17300,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":36565,"spinlock_wait_cycles":9390976,"thread_start_us":253778,"threads_started":2,"wal-append.queue_time_us":188790}
I20260807 08:01:39.996335 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:39.976178 17190 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:39.997819 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:39.998592 13804 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 6: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:39.976845 17431 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:39.999868 17557 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:39.999231 13657 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:39.920197 17548 consensus_queue.cc:1046] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
W20260807 08:01:40.008176 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 52151.2us
I20260807 08:01:39.939316 16976 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:39.989748 17303 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:39.997996 17543 log.cc:924] Time spent T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.198s user 0.000s sys 0.001s
I20260807 08:01:39.934303 17218 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.015388 13769 leader_election.cc:398] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:39.957758 17401 consensus_queue.cc:1046] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
W20260807 08:01:40.007710 13540 connection.cc:232] RPC call timeout handler was delayed by -1.4118217539507896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16190000000000002s for a response
W20260807 08:01:40.029145 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 52446.3us
I20260807 08:01:40.031091 13769 leader_election.cc:401] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260807 08:01:40.046159 13540 connection.cc:232] RPC call timeout handler was delayed by -1.3830217539798468s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16510000000000002s for a response
W20260807 08:01:40.046854 13540 connection.cc:232] RPC call timeout handler was delayed by -1.2390217538923025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
W20260807 08:01:40.047318 13540 connection.cc:232] RPC call timeout handler was delayed by -1.2336217539850622s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1817s for a response
W20260807 08:01:40.047670 13540 connection.cc:232] RPC call timeout handler was delayed by -1.1643217538949102s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
W20260807 08:01:40.048017 13540 connection.cc:232] RPC call timeout handler was delayed by -1.1409217540640384s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.192s for a response
W20260807 08:01:40.048620 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 49491.3us
I20260807 08:01:40.049784 13540 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:40.052659 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:40.008778 17421 raft_consensus.cc:686] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:40.014866 17557 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:40.060002 13657 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:40.060189 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:40.014508 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:40.013151 17390 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:40.063236 13540 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:40.070106 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 54837.7us
I20260807 08:01:40.053885 16791 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:39.989575 17449 consensus_queue.cc:237] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.078531 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.078707 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.053678 16459 raft_consensus.cc:686] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:40.061391 17431 raft_consensus.cc:3032] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:40.081152 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.083516 14627 raft_consensus.cc:1226] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 08:01:40.086228 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.085438 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.084465 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:40.087693 13580 raft_consensus.cc:2379] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:40.088575 13589 raft_consensus.cc:2379] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:40.089083 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.062330 17491 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:40.083156 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 103463us
I20260807 08:01:40.066599 17190 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:40.095839 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:40.064939 17557 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:40.076540 17401 raft_consensus.cc:2726] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:40.102401 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.074831 17447 raft_consensus.cc:686] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:40.055985 16992 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.084805 16791 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.075539 17390 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.066674 17527 consensus_queue.cc:1046] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.238s
I20260807 08:01:40.386241 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:40.387058 17585 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:40.387630 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.388257 13589 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:40.104879 17190 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:40.349197 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 247941us
I20260807 08:01:40.106707 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:40.389153 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:40.406764 13592 raft_consensus.cc:2362] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5. Re-sending same reply.
I20260807 08:01:40.387625 17585 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:40.154323 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.085598 17431 raft_consensus.cc:729] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:40.400161 17548 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:40.400454 17443 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:40.153971 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.075320 14620 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:40.403532 13769 connection.cc:232] RPC call timeout handler was delayed by -1.110107742017135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3178s for a response
I20260807 08:01:40.402204 17390 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:40.405884 13540 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:40.440047 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:40.441402 13540 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:40.401260 16791 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:40.148222 17438 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:40.453341 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.152014 17481 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:40.410872 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 350295us
I20260807 08:01:40.152673 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.146490 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:40.153565 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:40.380578 17442 consensus_meta.cc:287] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.715s user 0.010s sys 0.000s
I20260807 08:01:40.216307 17439 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:40.367116 17368 consensus_meta.cc:287] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.526s user 0.007s sys 0.003s
I20260807 08:01:40.304558 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:40.107497 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:40.152153 17569 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election won for term 5
W20260807 08:01:40.454058 17513 log.cc:924] Time spent T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.884s user 0.000s sys 0.005s
I20260807 08:01:40.103770 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:40.445024 17427 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:40.151216 17572 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260807 08:01:40.154456 17564 log.cc:924] Time spent T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
I20260807 08:01:40.456897 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.380775 17581 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:40.402748 16992 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:40.152302 17479 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:40.348184 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:40.469426 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:40.153893 17552 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:40.151644 17566 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:40.410485 17585 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:40.474102 13584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1773) took 1874 ms (client timeout 1504 ms). Trace:
I20260807 08:01:40.485810 13591 raft_consensus.cc:2362] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5. Re-sending same reply.
I20260807 08:01:40.083120 17554 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:40.453351 17538 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:40.151355 17522 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.394s
I20260807 08:01:40.154347 17489 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:40.495146 17368 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:40.469925 17218 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:40.474776 17401 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:40.678426 14620 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 08:01:40.678052 17401 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:40.584810 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 45954.3us
I20260807 08:01:40.681634 17480 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:40.682906 17571 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:40.683632 17571 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.684585 13769 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:40.686105 13811 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
W20260807 08:01:40.686792 17583 log.cc:924] Time spent T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
W20260807 08:01:40.687168 13708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1746) took 1692 ms (client timeout 1917 ms). Trace:
W20260807 08:01:40.687672 13708 rpcz_store.cc:269] 0807 08:01:38.995134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.242026 (+246892us) service_pool.cc:227] Handling call
0807 08:01:40.686927 (+1444901us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b5400322654. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:40.687092 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52688128}
I20260807 08:01:40.686028 17401 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:40.490566 17439 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.688853 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:40.690013 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 683) took 2419 ms (client timeout 2160 ms). Trace:
W20260807 08:01:40.690500 14626 rpcz_store.cc:269] 0807 08:01:38.270497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.270627 (+ 130us) service_pool.cc:227] Handling call
0807 08:01:40.689812 (+2419185us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54003309d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:40.689926 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1142196480}
I20260807 08:01:40.691109 17439 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:40.691043 13769 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:40.691962 17439 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:40.692811 17439 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:40.474300 17481 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.694720 13769 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:40.697953 17545 log.cc:924] Time spent T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
W20260807 08:01:40.698174 13706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 378) took 2290 ms (client timeout 2186 ms). Trace:
W20260807 08:01:40.698777 13706 rpcz_store.cc:269] 0807 08:01:38.408016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.639798 (+231782us) service_pool.cc:227] Handling call
0807 08:01:40.697916 (+2058118us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b540039d854. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:40.698056 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1957077248}
I20260807 08:01:40.379295 17477 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:40.586392 13584 rpcz_store.cc:269] 0807 08:01:38.599755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.599958 (+ 203us) service_pool.cc:227] Handling call
0807 08:01:40.466482 (+1866524us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400076f54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:40.474008 (+ 7526us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2393920768}
I20260807 08:01:40.151003 16459 consensus_queue.cc:237] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.456413 17540 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.129s
I20260807 08:01:40.380306 17397 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.664482 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:40.724689 14629 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:40.541738 14623 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:40.524035 17569 raft_consensus.cc:686] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:40.485191 17390 consensus_queue.cc:1046] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.102s
I20260807 08:01:40.466816 17438 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:40.675704 13538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.000s sys 0.010s
W20260807 08:01:40.775054 13538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.869s user 0.000s sys 0.011s
I20260807 08:01:40.777185 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.523595 17427 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:40.663595 17554 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:40.696358 17481 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:40.700202 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:40.700579 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 383) took 2253 ms. Trace:
I20260807 08:01:40.522890 17597 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.243s
I20260807 08:01:40.350322 17421 consensus_queue.cc:237] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.704268 13769 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:40.439821 17548 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:40.712589 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 304613us
W20260807 08:01:40.848143 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 143958us
I20260807 08:01:40.878909 17593 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:40.922492 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:40.921687 17593 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.874652 17475 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:40.719235 13701 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:40.589181 17552 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:40.549485 17581 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.923702 17552 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:40.924120 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:40.924866 13701 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:40.926589 13701 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 08:01:40.926947 17581 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:40.928000 17581 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:40.873737 17572 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:40.677196 17570 log.cc:924] Time spent T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.252s user 0.001s sys 0.000s
W20260807 08:01:40.722066 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 57637.8us
I20260807 08:01:40.848050 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1775) took 2186 ms. Trace:
I20260807 08:01:40.928633 17581 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:40.676139 17489 consensus_queue.cc:1035] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:40.778009 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.723846 17439 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:40.799592 13710 rpcz_store.cc:276] 0807 08:01:38.446905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.632193 (+1185288us) service_pool.cc:227] Handling call
0807 08:01:39.632548 (+ 355us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:39.656779 (+ 24231us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:39.656811 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:39.673170 (+ 16359us) log.cc:842] Serialized 26 byte log entry
0807 08:01:39.879716 (+206546us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:39.900094 (+ 20378us) log.cc:842] Serialized 14 byte log entry
0807 08:01:39.900552 (+ 458us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:39.900896 (+ 344us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:40.394246 (+493350us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b18003f2aa0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:40.700073 (+305827us) raft_consensus.cc:1671] finished
0807 08:01:40.700325 (+ 252us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:40.700504 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29789,"prepare.queue_time_us":20135,"raft.queue_time_us":28170,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":12252,"spinlock_wait_cycles":6418560,"thread_start_us":178051,"threads_started":1,"wal-append.queue_time_us":231311}
I20260807 08:01:40.440054 17443 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.805212 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:40.838313 13706 raft_consensus.cc:2379] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:40.798619 17538 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:40.931238 17600 log.cc:924] Time spent T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.204s user 0.001s sys 0.000s
I20260807 08:01:40.844677 14622 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 6: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:40.669144 17431 consensus_queue.cc:260] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:40.658526 17190 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:41.004057 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.730969 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:40.938448 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:40.929059 17572 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.666404 17447 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:41.011868 17438 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:41.012336 17443 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:41.013026 17605 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:41.013855 17605 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:40.838513 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:40.945016 17475 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.815382 17449 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:41.016316 17423 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:41.016290 17449 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:40.998212 17440 consensus_meta.cc:287] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 1.039s user 0.009s sys 0.004s
I20260807 08:01:41.018368 17475 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:41.018234 17440 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:41.018963 17397 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:40.998340 17538 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:41.021575 17569 consensus_queue.cc:237] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.895035 17611 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:41.010630 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 72260.8us
W20260807 08:01:40.936043 17367 consensus_meta.cc:287] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.976s user 0.008s sys 0.005s
I20260807 08:01:40.881320 17592 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:40.877759 17596 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:40.881953 17586 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:40.938753 17477 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:41.017210 17423 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:41.021720 13580 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:40.963075 13661 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:40.938985 17597 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:40.798508 17554 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:41.021010 17449 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:41.111692 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:41.113072 14620 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:41.058681 17605 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:41.042455 14583 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:41.095984 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:40.963590 17303 consensus_meta.cc:287] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.906s user 0.010s sys 0.001s
I20260807 08:01:40.947013 17218 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:41.003638 17557 consensus_meta.cc:287] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.858s user 0.009s sys 0.000s
I20260807 08:01:41.051318 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:41.101953 17443 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:40.948043 17481 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:40.878396 17485 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.326s
W20260807 08:01:41.099231 13659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.019s sys 0.053s
I20260807 08:01:41.096463 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:41.133916 13659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.549s user 0.019s sys 0.053s
I20260807 08:01:40.954603 17439 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:40.930845 13585 rpcz_store.cc:276] 0807 08:01:38.614917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.615303 (+ 386us) service_pool.cc:227] Handling call
0807 08:01:38.617632 (+ 2329us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:38.636993 (+ 19361us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:38.637035 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:38.659487 (+ 22452us) log.cc:842] Serialized 52 byte log entry
0807 08:01:39.019494 (+360007us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:39.510160 (+490666us) log.cc:842] Serialized 14 byte log entry
0807 08:01:39.848689 (+338529us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b18003cfee0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b19fba4ca 00007f6b2369bf99 00007f6b19e7e990 00007f6b1a040864 00007f6b1a040155 00007f6b20940167 00007f6b1cf75807 00007f6b1cf84f0f 00007f6b1cf84e88
0807 08:01:39.848895 (+ 206us) log.cc:842] Serialized 14 byte log entry
0807 08:01:39.849066 (+ 171us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:39.849137 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:40.156947 (+307810us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:40.800929 (+643982us) raft_consensus.cc:1671] finished
0807 08:01:40.800957 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:40.801100 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":758469,"prepare.queue_time_us":453958,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":4322,"raft.queue_time_us":220009,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":1832,"spinlock_wait_cycles":30195456,"thread_start_us":242511,"threads_started":1,"wal-append.queue_time_us":789848}
I20260807 08:01:41.024505 17538 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:41.037763 17431 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:41.084200 17367 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:41.156700 17602 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:41.157495 17602 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:41.131910 17443 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:41.097994 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:41.043006 17190 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:41.122578 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 80265.8us
I20260807 08:01:41.172717 14583 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:41.124162 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 161183us
I20260807 08:01:41.127188 17303 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:40.954428 17550 raft_consensus.cc:3032] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:41.036872 17429 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:41.103899 17440 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:41.121146 17597 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:41.110415 17423 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:41.063805 17611 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:41.133306 13653 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.530s user 0.024s sys 0.064s
I20260807 08:01:41.106897 17540 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:41.141250 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:41.097056 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
W20260807 08:01:41.191027 14583 connection.cc:232] RPC call timeout handler was delayed by -1.0792044419795275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
I20260807 08:01:41.136571 17481 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:41.108435 17586 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:41.145243 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:41.116151 17571 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:41.133671 17557 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:41.154310 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:41.105234 17596 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:41.109637 14629 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:41.142228 17439 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:41.124423 13540 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:41.179296 13703 raft_consensus.cc:2379] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:41.181941 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:41.170972 17443 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:41.183337 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:41.185216 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:41.185312 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:41.185658 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:41.186605 13661 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:41.153295 17607 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:41.187902 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:41.130586 17218 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:41.180248 17190 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:41.187887 17550 raft_consensus.cc:729] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:41.169445 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:41.100459 17592 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:41.035946 17489 consensus_queue.cc:1046] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260807 08:01:41.192776 17548 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:41.194036 17572 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:41.193038 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:41.205535 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:41.191617 14623 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:41.206656 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:41.025228 17401 consensus_queue.cc:1159] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "77da774f2bb54cf28302d53d0db224ad" responder_term: 6 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.4-5.4]" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 08:01:41.234072 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:41.470897 13657 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.012s sys 0.021s
W20260807 08:01:42.166747 13657 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.319s user 0.017s sys 0.029s
I20260807 08:01:41.199697 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:41.693919 13590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:39822 (request call id 17) took 2885 ms. Trace:
I20260807 08:01:41.274487 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:42.036710 13702 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.004s sys 0.005s
W20260807 08:01:42.152200 13659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.016s user 0.007s sys 0.016s
I20260807 08:01:41.299614 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:42.166049 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 690) took 1583 ms. Trace:
I20260807 08:01:42.165568 13580 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:41.186640 17614 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:42.248185 13811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 526) took 1698 ms (client timeout 2022 ms). Trace:
I20260807 08:01:42.168376 13588 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
W20260807 08:01:42.167466 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 980845us
W20260807 08:01:42.248857 13811 rpcz_store.cc:269] 0807 08:01:40.468209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:40.469124 (+ 915us) service_pool.cc:227] Handling call
0807 08:01:42.166140 (+1697016us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b540027b0d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:42.166237 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":314674048}
I20260807 08:01:42.168581 13708 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
W20260807 08:01:42.172672 13702 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.995s user 0.006s sys 0.006s
I20260807 08:01:42.250128 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 386) took 1065 ms. Trace:
I20260807 08:01:42.250479 13708 rpcz_store.cc:276] 0807 08:01:41.184474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.184833 (+ 359us) service_pool.cc:227] Handling call
0807 08:01:42.250072 (+1065239us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:41.319108 14580 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.006s sys 0.009s
I20260807 08:01:42.171523 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:42.243783 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1795) took 1062 ms. Trace:
I20260807 08:01:41.207901 14627 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:42.249451 13588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33050 (request call id 513) took 1062 ms. Trace:
I20260807 08:01:42.168202 14625 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
W20260807 08:01:42.247887 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 1.01383e+06us
W20260807 08:01:42.173938 13659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.038s user 0.008s sys 0.016s
W20260807 08:01:42.251430 14580 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.519s user 0.014s sys 0.021s
I20260807 08:01:42.252256 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 532) took 1072 ms. Trace:
W20260807 08:01:42.252465 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 1.51989e+06us
I20260807 08:01:42.167557 13590 rpcz_store.cc:276] 0807 08:01:38.808236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:38.808403 (+ 167us) service_pool.cc:227] Handling call
0807 08:01:41.693702 (+2885299us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b540036a954. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf65bc3 00007f6b21eaf81a 00007f6b1b5a2d04 00007f6b1b5a2c92 00007f6b1b5a2bc1 00007f6b1b5a2b3c 00007f6b1b5a1df2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:41.693812 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":438808960}
I20260807 08:01:42.244035 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33050 (request call id 514) took 1045 ms. Trace:
I20260807 08:01:42.252012 13588 rpcz_store.cc:276] 0807 08:01:41.187473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.187618 (+ 145us) service_pool.cc:227] Handling call
0807 08:01:42.249391 (+1061773us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:42.253538 13592 rpcz_store.cc:276] 0807 08:01:41.198978 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.199095 (+ 117us) service_pool.cc:227] Handling call
0807 08:01:42.243978 (+1044883us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:42.251869 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 536) took 1153 ms. Trace:
I20260807 08:01:42.244207 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1794) took 1063 ms. Trace:
I20260807 08:01:42.254796 13585 rpcz_store.cc:276] 0807 08:01:41.181117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.182955 (+ 1838us) service_pool.cc:227] Handling call
0807 08:01:42.244156 (+1061201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:42.251700 13580 rpcz_store.cc:276] 0807 08:01:41.181471 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.181628 (+ 157us) service_pool.cc:227] Handling call
0807 08:01:42.243701 (+1062073us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:42.167958 13706 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:01:42.256898 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 387) took 1072 ms. Trace:
I20260807 08:01:42.257241 13706 rpcz_store.cc:276] 0807 08:01:41.184829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.184992 (+ 163us) service_pool.cc:227] Handling call
0807 08:01:42.256832 (+1071840us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:42.172429 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 535) took 1074 ms. Trace:
I20260807 08:01:42.258173 14626 rpcz_store.cc:276] 0807 08:01:41.097590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.134123 (+ 36533us) service_pool.cc:227] Handling call
0807 08:01:42.172355 (+1038232us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:42.192744 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 1.34531e+06us
I20260807 08:01:42.252616 14625 rpcz_store.cc:276] 0807 08:01:41.180007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.199846 (+ 19839us) service_pool.cc:227] Handling call
0807 08:01:42.252201 (+1052355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:42.263729 14583 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:42.245970 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33050 (request call id 515) took 1005 ms. Trace:
I20260807 08:01:42.254676 14627 rpcz_store.cc:276] 0807 08:01:41.098078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.136386 (+ 38308us) service_pool.cc:227] Handling call
0807 08:01:42.251815 (+1115429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:42.265399 13587 rpcz_store.cc:276] 0807 08:01:41.240338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.240554 (+ 216us) service_pool.cc:227] Handling call
0807 08:01:42.245908 (+1005354us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:41.729961 17423 consensus_meta.cc:287] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.535s user 0.013s sys 0.000s
I20260807 08:01:42.286814 17423 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.267360 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:42.290091 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 390) took 1104 ms. Trace:
I20260807 08:01:42.290493 13710 rpcz_store.cc:276] 0807 08:01:41.185853 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:42.166546 (+980693us) service_pool.cc:227] Handling call
0807 08:01:42.290021 (+123475us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:41.732357 17554 consensus_meta.cc:287] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.620s user 0.011s sys 0.003s
I20260807 08:01:42.291589 17554 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:41.203068 17486 consensus_meta.cc:287] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.318s user 0.011s sys 0.003s
I20260807 08:01:42.293874 17486 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.297930 13705 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:41.231372 17599 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:42.299010 17599 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:41.337124 17524 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.016s sys 0.128s
W20260807 08:01:42.302060 17524 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.626s user 0.028s sys 0.157s
I20260807 08:01:42.303041 17524 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:42.303757 17524 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:42.304361 17524 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:42.268151 17614 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:41.196930 17597 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:42.243170 17568 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 08:01:41.710973 17552 consensus_meta.cc:287] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.786s user 0.012s sys 0.003s
I20260807 08:01:42.243664 17449 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 08:01:41.711839 17585 consensus_meta.cc:287] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.999s user 0.013s sys 0.000s
I20260807 08:01:41.229106 14618 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
W20260807 08:01:41.429493 17566 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.035s sys 0.052s
I20260807 08:01:41.226014 17609 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:42.185432 14629 rpcz_store.cc:276] 0807 08:01:40.583163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:40.583340 (+ 177us) service_pool.cc:227] Handling call
0807 08:01:42.165983 (+1582643us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:41.712620 16976 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:42.257712 14623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 687) took 2171 ms (client timeout 1778 ms). Trace:
I20260807 08:01:42.247759 14620 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:42.278340 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:42.279541 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 534) took 1182 ms. Trace:
I20260807 08:01:42.280143 17190 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:42.406556 14626 rpcz_store.cc:276] 0807 08:01:41.097204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:42.259172 (+1161968us) service_pool.cc:227] Handling call
0807 08:01:42.279019 (+ 19847us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:42.279227 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:42.279249 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:42.279305 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:42.279334 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:42.279397 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:42.279492 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:42.280488 17431 consensus_meta.cc:287] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 1.117s user 0.011s sys 0.002s
I20260807 08:01:42.283672 13591 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:41.707409 17367 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:41.291508 17612 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.994s
I20260807 08:01:42.285243 17550 consensus_queue.cc:260] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:41.206185 17540 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.272369 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1483) took 2421 ms. Trace:
I20260807 08:01:41.543833 17603 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:41.299165 17478 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:42.151958 17499 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.991s user 0.009s sys 0.015s
I20260807 08:01:42.316392 14630 raft_consensus.cc:1226] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:42.377810 17423 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:42.380874 14625 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:42.380156 17599 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:42.314806 17607 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:42.315456 17552 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:42.165187 17429 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:41.228098 17438 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:42.430891 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 530) took 1329 ms. Trace:
I20260807 08:01:41.567006 17617 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:42.391616 17586 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.376138 17554 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:41.230285 17303 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:41.428948 17605 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:41.296988 17616 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:41.228519 17611 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:41.297104 17440 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:41.315246 17581 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:41.558851 17387 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:41.434495 17593 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.283123 17218 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:42.408722 13708 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:42.493906 17481 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:42.494285 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 391) took 1308 ms. Trace:
I20260807 08:01:42.494670 13708 rpcz_store.cc:276] 0807 08:01:41.186107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:42.267121 (+1081014us) service_pool.cc:227] Handling call
0807 08:01:42.494224 (+227103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:42.495774 17557 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:42.484664 17303 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:42.502666 17578 raft_consensus.cc:3032] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:42.503391 17578 raft_consensus.cc:729] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:42.505987 17439 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:42.510200 13705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1477) took 2708 ms (client timeout 2131 ms). Trace:
W20260807 08:01:42.510744 13705 rpcz_store.cc:269] 0807 08:01:39.801535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.809611 (+ 8076us) service_pool.cc:227] Handling call
0807 08:01:42.509894 (+2700283us) spinlock_profiling.cc:243] Waited 2.41 s on lock 0x7b5400407c54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:42.510065 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5537110784}
I20260807 08:01:42.510326 17439 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:42.517079 13584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1792) took 1561 ms. Trace:
I20260807 08:01:42.517769 13584 rpcz_store.cc:276] 0807 08:01:40.955631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:40.955762 (+ 131us) service_pool.cc:227] Handling call
0807 08:01:40.956079 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:42.337176 (+1381097us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:42.337218 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:42.337284 (+ 66us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:42.337347 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:42.516638 (+179291us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540025dd54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:42.516710 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:42.516897 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3175663488}
I20260807 08:01:42.517150 17524 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.525092 17524 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.297345 17486 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.264875 17571 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:42.316509 14619 raft_consensus.cc:1226] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
W20260807 08:01:42.394295 17566 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.668s user 0.044s sys 0.056s
W20260807 08:01:42.152335 17508 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.983s user 0.004s sys 0.018s
I20260807 08:01:42.313469 17614 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:42.421873 13769 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.492s user 0.012s sys 0.016s
I20260807 08:01:42.419322 13702 rpcz_store.cc:276] 0807 08:01:39.851165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.865138 (+ 13973us) service_pool.cc:227] Handling call
0807 08:01:39.865452 (+ 314us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:39.877447 (+ 11995us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:39.877487 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:39.885127 (+ 7640us) log.cc:842] Serialized 52 byte log entry
0807 08:01:39.978275 (+ 93148us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:41.177147 (+1198872us) spinlock_profiling.cc:243] Waited 6.64 ms on lock 0x7b18002d6f40. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1cbfa82c 00007f6b1a0259dc 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a040155 00007f6b20940167 00007f6b1cf75807 00007f6b1cf84f0f
0807 08:01:41.177300 (+ 153us) log.cc:842] Serialized 14 byte log entry
0807 08:01:42.250424 (+1073124us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:42.250507 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:42.250554 (+ 47us) raft_consensus.cc:1671] finished
0807 08:01:42.272069 (+ 21515us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:42.272253 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":14914,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":55,"raft.queue_time_us":166712,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":10433,"spinlock_wait_cycles":15281024,"thread_start_us":1187289,"threads_started":3,"wal-append.queue_time_us":79782,"wal-append.run_cpu_time_us":2630,"wal-append.run_wall_time_us":1175654}
I20260807 08:01:42.416443 17190 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:42.415231 17548 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:42.401820 14623 rpcz_store.cc:269] 0807 08:01:40.076588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:40.081396 (+ 4808us) service_pool.cc:227] Handling call
0807 08:01:42.247645 (+2166249us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:42.426038 17499 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.264s user 0.012s sys 0.019s
I20260807 08:01:42.393113 17585 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:42.427416 17421 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.403s user 0.008s sys 0.021s
I20260807 08:01:42.436252 14625 rpcz_store.cc:276] 0807 08:01:41.101191 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:42.260857 (+1159666us) service_pool.cc:227] Handling call
0807 08:01:42.372750 (+111893us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:42.430565 (+ 57815us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:42.430821 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:42.438282 13701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.251s user 0.006s sys 0.023s
W20260807 08:01:42.439886 13653 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.242s user 0.039s sys 0.048s
I20260807 08:01:42.431754 17367 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:42.410771 17609 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:42.437120 16459 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.514s user 0.029s sys 0.013s
I20260807 08:01:42.425719 17478 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:42.490360 13768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.001s user 0.013s sys 0.017s
W20260807 08:01:42.490836 14579 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.890s user 0.001s sys 0.026s
W20260807 08:01:42.491181 14582 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.963s user 0.012s sys 0.030s
I20260807 08:01:42.481719 17605 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:42.485015 17616 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.492655 17592 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.493347 17596 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:42.244937 17602 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.534631 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:42.535683 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:42.541080 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 277441us
W20260807 08:01:42.632826 14582 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.125s user 0.012s sys 0.030s
I20260807 08:01:42.568356 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:42.438889 17429 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:42.555562 13769 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.625s user 0.012s sys 0.016s
W20260807 08:01:42.637456 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 1.7164e+06us
W20260807 08:01:42.542526 17447 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.527s user 0.007s sys 0.031s
I20260807 08:01:42.441473 17438 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.528563 17387 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.633352 13705 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
W20260807 08:01:42.634516 13768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.211s user 0.013s sys 0.017s
I20260807 08:01:42.520744 17443 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:42.634804 17421 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.610s user 0.008s sys 0.021s
I20260807 08:01:42.678344 13584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39822
W20260807 08:01:42.635437 13701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.449s user 0.006s sys 0.024s
W20260807 08:01:42.635804 14579 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.098s user 0.001s sys 0.027s
W20260807 08:01:42.636022 13653 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.438s user 0.039s sys 0.049s
W20260807 08:01:42.636123 14583 connection.cc:232] RPC call timeout handler was delayed by -1.1169713630806655s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2062s for a response
I20260807 08:01:42.636523 13809 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 6: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:42.638700 17367 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:42.681213 17639 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:42.639791 17447 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.624s user 0.007s sys 0.034s
I20260807 08:01:42.627507 17524 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:42.540517 17439 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:42.638674 13538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.875s user 0.054s sys 0.052s
W20260807 08:01:42.563505 17508 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.394s user 0.004s sys 0.019s
W20260807 08:01:42.550884 17544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.760s user 0.023s sys 0.018s
W20260807 08:01:42.549240 17569 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.351s user 0.012s sys 0.027s
I20260807 08:01:42.700064 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
W20260807 08:01:42.638413 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 378435us
I20260807 08:01:42.488161 17440 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:42.636554 16459 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.714s user 0.029s sys 0.014s
I20260807 08:01:42.436030 17607 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.623432 17242 raft_consensus.cc:3032] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:42.564085 17566 raft_consensus.cc:686] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:42.563692 17190 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:42.627852 17481 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:42.633286 13811 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:42.635360 17218 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:42.504899 17578 consensus_queue.cc:260] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.472676 17617 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.687618 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:42.689695 13538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.930s user 0.054s sys 0.053s
I20260807 08:01:42.684087 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:42.681382 17645 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260807 08:01:42.685654 17639 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:42.702499 17569 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.502s user 0.013s sys 0.028s
I20260807 08:01:42.774999 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1500) took 3486 ms. Trace:
I20260807 08:01:42.779861 13814 rpcz_store.cc:276] 0807 08:01:39.259965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.260142 (+ 177us) service_pool.cc:227] Handling call
0807 08:01:39.263885 (+ 3743us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:39.796833 (+532948us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:42.741386 (+2944553us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b540027c4d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:42.745809 (+ 4423us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1219722112}
I20260807 08:01:42.780360 17438 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:42.774668 17635 log.cc:924] Time spent T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260807 08:01:42.783464 17440 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.784031 17617 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.787652 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:42.690857 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:42.812919 14623 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:42.813412 17478 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.743295 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 527) took 2072 ms. Trace:
W20260807 08:01:42.742708 13591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1787) took 1965 ms (client timeout 1775 ms). Trace:
W20260807 08:01:42.821318 13591 rpcz_store.cc:269] 0807 08:01:40.777077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:40.777696 (+ 619us) service_pool.cc:227] Handling call
0807 08:01:42.742540 (+1964844us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400369f54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:42.742640 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2915354368}
I20260807 08:01:42.823289 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:42.824061 13590 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:42.829380 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:42.829319 17607 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.830377 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:42.813356 17566 consensus_queue.cc:237] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:42.778915 17429 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.832830 17605 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:42.833666 17569 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:42.834331 17569 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:42.836851 14620 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:42.691596 17632 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:42.692392 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:42.862591 14626 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:42.692507 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:42.692658 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:42.687160 17581 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:42.553402 17571 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:42.698659 14583 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:42.878912 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 180307us
I20260807 08:01:42.695191 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:42.702047 13657 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:42.881356 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 179390us
W20260807 08:01:42.688472 17606 log.cc:924] Time spent T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 1.798s user 0.012s sys 0.018s
I20260807 08:01:42.574671 17548 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:42.682195 17641 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:42.792421 17387 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.798681 13705 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:42.906375 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:42.907179 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:42.733301 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:42.732882 17601 log.cc:924] Time spent T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.200s user 0.001s sys 0.001s
I20260807 08:01:42.805136 17190 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:42.691133 17552 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:42.697851 17544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.952s user 0.023s sys 0.020s
I20260807 08:01:42.856786 14619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 532) took 1760 ms. Trace:
I20260807 08:01:42.687534 17486 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:42.819537 13808 rpcz_store.cc:276] 0807 08:01:40.671370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:40.671580 (+ 210us) service_pool.cc:227] Handling call
0807 08:01:40.671934 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:40.701483 (+ 29549us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:40.701558 (+ 75us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:40.701597 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:40.701626 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:42.743021 (+2041395us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b54003c3cd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:42.743083 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:42.743220 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67450240}
I20260807 08:01:43.011603 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:42.690181 13769 connection.cc:232] RPC call timeout handler was delayed by -2.2888426338322461s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1917s for a response
I20260807 08:01:42.873832 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:42.875402 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:42.895031 17478 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:42.876942 13589 raft_consensus.cc:1226] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260807 08:01:42.876994 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:43.016573 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:42.732998 13700 raft_consensus.cc:2379] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:42.863679 14625 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:42.887024 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.019325 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.020018 13811 raft_consensus.cc:2396] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260807 08:01:43.020150 13808 raft_consensus.cc:2396] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260807 08:01:43.017050 13769 connection.cc:232] RPC call timeout handler was delayed by -1.25267845694907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1512s for a response
W20260807 08:01:43.020817 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1734784569125623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16000000000000003s for a response
W20260807 08:01:43.021231 13769 connection.cc:232] RPC call timeout handler was delayed by -1.0717784569133073s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
I20260807 08:01:42.776551 17609 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.837304 17569 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.895277 13710 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
W20260807 08:01:42.898200 14583 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 2.062s (SENT) [suppressed 17 similar messages]
I20260807 08:01:43.025874 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:42.851146 17242 raft_consensus.cc:729] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:42.896299 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:42.900820 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 349872us
I20260807 08:01:42.741561 17630 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:42.889710 13540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.006s sys 0.012s
W20260807 08:01:43.034348 13540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.864s user 0.006s sys 0.012s
I20260807 08:01:43.034348 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:43.034691 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 1.91026e+06us
I20260807 08:01:42.907976 13705 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:42.815835 17639 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:42.889811 17581 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:42.694911 17654 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:43.004097 14580 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.032s sys 0.064s
I20260807 08:01:42.695948 17524 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.009011 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:42.894328 17596 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:42.860994 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:42.887645 14620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 533) took 1777 ms (client timeout 1569 ms). Trace:
I20260807 08:01:43.012851 13579 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:42.875799 17573 raft_consensus.cc:2726] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:43.010807 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.012454 13706 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:42.815550 17218 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:42.987998 17619 log.cc:924] Time spent T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.185s user 0.000s sys 0.000s
I20260807 08:01:43.017107 13592 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:42.840989 17481 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:42.888264 13590 raft_consensus.cc:2396] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260807 08:01:43.021667 13769 connection.cc:232] RPC call timeout handler was delayed by -1.0132784568704665s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
I20260807 08:01:42.702950 17439 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.024433 13584 raft_consensus.cc:595] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:43.027007 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:43.028548 13814 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:43.030611 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:43.032444 13661 leader_election.cc:334] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.504s (SENT)
I20260807 08:01:43.035110 13589 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:42.878180 17632 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:42.987833 17641 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.035724 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:43.114892 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.092232 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1488) took 3236 ms. Trace:
I20260807 08:01:43.079046 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:43.129031 13710 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:43.129679 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:43.078110 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.082161 13587 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 08:01:43.130695 14626 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:43.127794 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 95374.5us
I20260807 08:01:43.131561 13808 raft_consensus.cc:2379] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 6: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:43.132449 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.132619 13661 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.133196 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.127516 13817 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:43.098804 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.045306 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.135500 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:43.137444 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:43.137380 13579 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:43.128069 13709 rpcz_store.cc:276] 0807 08:01:39.855987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.974622 (+118635us) service_pool.cc:227] Handling call
0807 08:01:39.991119 (+ 16497us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:39.991410 (+ 291us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:39.991438 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:39.991831 (+ 393us) log.cc:842] Serialized 26 byte log entry
0807 08:01:40.456075 (+464244us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:40.489212 (+ 33137us) log.cc:842] Serialized 14 byte log entry
0807 08:01:40.489388 (+ 176us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:40.489469 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:42.523469 (+2034000us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b18003401e0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:43.091877 (+568408us) raft_consensus.cc:1671] finished
0807 08:01:43.092000 (+ 123us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:43.092161 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5621,"prepare.queue_time_us":37163,"prepare.run_cpu_time_us":281,"prepare.run_wall_time_us":10229,"raft.queue_time_us":375065,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":69859,"spinlock_wait_cycles":76571008,"thread_start_us":395029,"threads_started":1,"wal-append.queue_time_us":635980}
I20260807 08:01:43.136174 13769 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:43.147467 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:43.147771 14580 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.881s user 0.032s sys 0.067s
I20260807 08:01:43.148231 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:43.148304 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:43.146456 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.135603 13804 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:43.151567 17617 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:43.138322 17242 consensus_queue.cc:260] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:43.152195 17617 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:43.152741 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.153689 13811 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:43.154012 13705 raft_consensus.cc:2379] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:43.151859 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:43.018085 17605 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:43.154930 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:43.158855 13807 raft_consensus.cc:2396] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 08:01:43.159150 13705 raft_consensus.cc:2379] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:43.160009 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:43.153246 17499 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:43.162446 13585 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:43.131019 17596 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:43.163676 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:43.156965 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:43.173825 13710 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:43.037304 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.164729 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:43.166318 13591 raft_consensus.cc:2396] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 08:01:43.021163 17486 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.175827 14580 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:43.175597 13589 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:43.168479 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:43.163600 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.036095 17593 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:43.179491 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.179766 13769 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.181761 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.152793 17242 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:43.152766 17617 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:43.185606 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:43.194741 13540 connection.cc:232] RPC call timeout handler was delayed by -1.3207620719913393s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21309999999999998s for a response
I20260807 08:01:43.194705 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
W20260807 08:01:43.195432 13540 connection.cc:232] RPC call timeout handler was delayed by -1.3207620719913393s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21309999999999998s for a response
I20260807 08:01:43.195704 14626 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:43.195789 13540 connection.cc:232] RPC call timeout handler was delayed by -1.1750321621075273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1559s for a response
W20260807 08:01:43.196175 13540 connection.cc:232] RPC call timeout handler was delayed by -1.1381321621593088s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16000000000000003s for a response
W20260807 08:01:43.196487 13540 connection.cc:232] RPC call timeout handler was delayed by -1.1381321621593088s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16000000000000003s for a response
I20260807 08:01:43.193145 13706 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
W20260807 08:01:43.196847 13540 connection.cc:232] RPC call timeout handler was delayed by -1.0912620720919222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23860000000000003s for a response
W20260807 08:01:43.197181 13540 connection.cc:232] RPC call timeout handler was delayed by -1.03726447513327s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
I20260807 08:01:43.198524 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:43.198853 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:43.198901 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:43.199990 13708 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
W20260807 08:01:43.127331 14620 rpcz_store.cc:269] 0807 08:01:41.096878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.128874 (+ 31996us) service_pool.cc:227] Handling call
0807 08:01:42.874606 (+1745732us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:43.163097 17645 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:43.216717 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:43.200454 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:43.202096 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:43.219262 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:43.220780 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:43.202234 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:43.222131 13590 raft_consensus.cc:2379] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:43.223218 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:43.224973 13590 raft_consensus.cc:1226] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260807 08:01:43.071149 17630 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.216265 13592 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:43.226105 13702 raft_consensus.cc:1226] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:43.175623 17596 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:43.183733 17429 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:43.201349 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:43.235982 14629 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:43.237533 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 339333us
I20260807 08:01:43.219866 17581 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:43.012866 14619 rpcz_store.cc:276] 0807 08:01:41.096278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.204757 (+108479us) service_pool.cc:227] Handling call
0807 08:01:42.257657 (+1052900us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:42.555485 (+297828us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:42.855572 (+300087us) spinlock_profiling.cc:243] Waited 49.1 ms on lock 0x7b54006734d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:42.856474 (+ 902us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b54006734d0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:42.856694 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2320936576}
I20260807 08:01:43.194414 17655 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:43.162683 17605 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:43.239873 14583 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:43.209977 17602 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:43.255894 13585 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:43.242084 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.185802 17609 consensus_queue.cc:1046] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260807 08:01:43.252835 14629 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:43.279320 14583 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:43.258301 14626 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:43.240863 13657 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:43.227243 17387 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:43.226384 17440 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:43.161617 17654 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.175136 17499 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.276120 17429 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:43.276182 13807 raft_consensus.cc:2396] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260807 08:01:43.217669 17571 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:43.236750 17508 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260807 08:01:43.323022 13709 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:01:43.281008 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 95501.2us
I20260807 08:01:43.282814 13657 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:43.242621 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 58087.6us
I20260807 08:01:43.248759 17566 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:43.188539 17593 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:43.275064 17602 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:43.275199 17620 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:43.253631 17655 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:43.288643 17440 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:43.374123 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 175679us
W20260807 08:01:43.377804 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 95117.6us
I20260807 08:01:43.412263 13657 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.412374 13769 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:43.378172 13661 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.413410 13657 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.376411 17447 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:43.290074 17387 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.377177 17566 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:43.415946 13709 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:01:43.323488 17678 raft_consensus.cc:2741] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 08:01:43.393328 17602 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:43.418639 13661 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:43.423496 13769 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:43.408612 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 120459us
I20260807 08:01:43.388645 17438 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 08:01:43.405923 17664 log.cc:924] Time spent T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.252s user 0.000s sys 0.002s
I20260807 08:01:43.384492 14580 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:43.429647 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.408177 17655 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:43.419719 17440 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:01:43.445258 13415 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet b09437796639470caac9548c71d72fb2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 7a889001d6494fe5b7cbbe7da1dae302 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:43.442268 13807 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:43.430116 17571 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 08:01:43.450477 17671 log.cc:924] Time spent T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260807 08:01:43.453776 17486 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:43.455812 17387 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:43.456669 17677 log.cc:924] Time spent T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
I20260807 08:01:43.461019 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1832) took 3802 ms. Trace:
I20260807 08:01:43.463624 17593 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:43.424914 17652 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:43.437218 17552 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:01:43.464815 13587 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:43.424535 17439 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:43.431505 17566 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:43.421381 17679 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:43.424839 17681 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:43.476585 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1486) took 3621 ms. Trace:
I20260807 08:01:43.426234 17641 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:43.479117 13805 rpcz_store.cc:276] 0807 08:01:39.658858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.659098 (+ 240us) service_pool.cc:227] Handling call
0807 08:01:43.460294 (+3801196us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b5800004298. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:43.460501 (+ 207us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:43.460684 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:43.460709 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:43.460749 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:43.460788 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:43.460850 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:43.460955 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6049024}
I20260807 08:01:43.431746 17218 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:43.432449 17645 raft_consensus.cc:686] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:43.488083 17603 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:43.466012 13588 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0248ade936924e628c46b8de6252af1b"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56576
I20260807 08:01:43.443886 17524 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:43.518505 17397 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:43.470024 17438 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
W20260807 08:01:43.521454 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 74129.4us
I20260807 08:01:43.471330 17481 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:43.550009 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 131446us
I20260807 08:01:43.525436 13711 rpcz_store.cc:276] 0807 08:01:39.855389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.914180 (+ 58791us) service_pool.cc:227] Handling call
0807 08:01:39.922881 (+ 8701us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:39.923138 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 08:01:39.941007 (+ 17869us) log.cc:842] Serialized 14 byte log entry
0807 08:01:40.402665 (+461658us) spinlock_profiling.cc:243] Waited 9.62 ms on lock 0x7b180032bfc0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:43.459890 (+3057225us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b180032bfc0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1cbfa82c 00007f6b1a0259dc 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a042e22 00007f6b1cbec5bb 00007f6b1cbf2063 00007f6b1cbf257b
0807 08:01:43.476125 (+ 16235us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:43.476205 (+ 80us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:43.476249 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:43.476338 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:43.476505 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5972,"prepare.queue_time_us":28195,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":35,"raft.queue_time_us":27396,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":17378,"spinlock_wait_cycles":37620864,"thread_start_us":18007,"threads_started":1,"wal-append.queue_time_us":18352,"wal-append.run_cpu_time_us":3119,"wal-append.run_wall_time_us":1540691}
I20260807 08:01:43.505904 17652 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:43.408169 17620 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:43.437601 17632 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:43.591715 13588 raft_consensus.cc:595] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Received request to transfer leadership
W20260807 08:01:43.782985 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 398569us
W20260807 08:01:43.517203 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 289406us
I20260807 08:01:43.667331 13589 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:43.525203 17681 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:43.710062 13585 consensus_queue.cc:237] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:43.796643 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.794742 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.522648 17679 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:43.795797 13540 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.798398 13804 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
W20260807 08:01:43.591639 17190 consensus_meta.cc:287] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.578s user 0.013s sys 0.000s
W20260807 08:01:43.731988 17682 log.cc:924] Time spent T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
W20260807 08:01:43.714330 17628 log.cc:924] Time spent T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
W20260807 08:01:43.697916 17671 log.cc:924] Time spent T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20260807 08:01:43.795336 13769 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:43.855458 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 60181.7us
I20260807 08:01:43.802146 17683 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:43.832360 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.668041 17605 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:43.840284 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:43.772330 17429 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:43.793499 17652 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:43.754107 17696 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:43.732033 17654 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:43.596611 17630 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:43.719421 17427 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:43.801350 17681 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:43.802527 17439 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:43.856838 13769 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.807178 13540 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.830479 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.805130 17679 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:43.857859 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 391387us
I20260807 08:01:43.839308 17218 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:43.805378 17552 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:43.858383 17571 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:43.848889 17499 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:43.876147 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.630787 17645 consensus_queue.cc:237] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.857966 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:43.842775 17190 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:43.915516 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1501) took 4631 ms. Trace:
I20260807 08:01:43.634397 17603 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:43.919162 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 389) took 2733 ms. Trace:
W20260807 08:01:43.922322 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 115218us
W20260807 08:01:43.918638 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 61874.7us
I20260807 08:01:43.923501 14626 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:43.924039 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:43.922670 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 131335us
I20260807 08:01:43.875438 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.927305 13816 rpcz_store.cc:276] 0807 08:01:39.260288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.262968 (+ 2680us) service_pool.cc:227] Handling call
0807 08:01:39.337117 (+ 74149us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:39.795926 (+458809us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:43.891104 (+4095178us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b540027abd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:43.891363 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1047870720}
I20260807 08:01:43.929502 13701 rpcz_store.cc:276] 0807 08:01:41.185541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.186097 (+ 556us) service_pool.cc:227] Handling call
0807 08:01:41.186444 (+ 347us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:42.679474 (+1493030us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:42.679507 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:42.679968 (+ 461us) log.cc:842] Serialized 26 byte log entry
0807 08:01:43.012870 (+332902us) raft_consensus.cc:1631] Marking committed up to 9
0807 08:01:43.249853 (+236983us) log.cc:842] Serialized 14 byte log entry
0807 08:01:43.254632 (+ 4779us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:43.254991 (+ 359us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:43.918897 (+663906us) raft_consensus.cc:1671] finished
0807 08:01:43.918926 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:43.919086 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4487,"prepare.queue_time_us":372208,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":382443,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":27692,"thread_start_us":1979494,"threads_started":3,"wal-append.queue_time_us":553729}
I20260807 08:01:43.960898 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.965049 13701 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:43.959035 13657 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.928227 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.965219 13817 raft_consensus.cc:2396] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 08:01:43.924505 17654 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:43.958407 13769 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:43.960613 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.928122 14620 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:43.925530 17605 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:43.926659 17387 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:43.921625 17571 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:43.921434 17218 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:43.969321 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:43.962033 13661 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:43.947834 17438 raft_consensus.cc:686] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:43.974396 13811 raft_consensus.cc:2379] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:01:43.963173 17617 consensus_meta.cc:287] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.734s user 0.009s sys 0.003s
I20260807 08:01:43.980602 13706 raft_consensus.cc:1226] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 08:01:44.009835 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:44.011976 13701 raft_consensus.cc:2379] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:44.013257 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:43.982159 14626 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 08:01:43.984929 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1492) took 1335 ms. Trace:
I20260807 08:01:43.863875 17683 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.015486 13702 rpcz_store.cc:276] 0807 08:01:42.649654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:42.677627 (+ 27973us) service_pool.cc:227] Handling call
0807 08:01:42.677999 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:43.229750 (+551751us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:43.984646 (+754896us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b5400408154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:43.984853 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1260138880}
I20260807 08:01:44.017037 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.012403 17617 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.017961 14629 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260807 08:01:44.017174 17438 consensus_queue.cc:237] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.018365 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:44.023900 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.027940 13817 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:44.026772 13816 raft_consensus.cc:2379] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:43.923560 17552 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:44.028841 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 71113.5us
I20260807 08:01:43.945849 16459 consensus_queue.cc:1046] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260807 08:01:43.977617 17387 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:43.945761 17439 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:43.987229 17571 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:43.980664 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1499) took 1280 ms. Trace:
I20260807 08:01:43.987633 13661 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:44.039556 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 81445.5us
I20260807 08:01:44.045472 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:43.983269 17697 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:44.033241 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:44.070600 13701 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:44.014979 17630 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 08:01:44.035048 17596 consensus_meta.cc:287] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.796s user 0.008s sys 0.005s
I20260807 08:01:44.050653 17190 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:44.052834 17602 consensus_meta.cc:287] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.608s user 0.007s sys 0.003s
I20260807 08:01:43.893769 17696 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.011435 17141 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260807 08:01:44.056466 17566 consensus_meta.cc:287] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.504s user 0.004s sys 0.006s
I20260807 08:01:43.983217 17218 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:43.927078 17708 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.106s
I20260807 08:01:43.977384 17605 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:44.068481 13705 rpcz_store.cc:276] 0807 08:01:42.700743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.184058 (+483315us) service_pool.cc:227] Handling call
0807 08:01:43.184369 (+ 311us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:43.188792 (+ 4423us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:43.188839 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:43.200320 (+ 11481us) log.cc:842] Serialized 26 byte log entry
0807 08:01:43.291584 (+ 91264us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:43.291641 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:43.291735 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:43.929155 (+637420us) raft_consensus.cc:1671] finished
0807 08:01:43.929183 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:43.980588 (+ 51405us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6268,"raft.queue_time_us":13981,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":33831,"thread_start_us":36664,"threads_started":1,"wal-append.queue_time_us":212177}
I20260807 08:01:44.080360 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:44.083012 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:44.022735 17429 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:44.022392 17427 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:44.080753 17697 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:44.086768 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 99369.2us
I20260807 08:01:44.087949 17713 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:44.088131 17596 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:44.036972 17440 consensus_meta.cc:287] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.547s user 0.009s sys 0.002s
W20260807 08:01:44.093469 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 146470us
I20260807 08:01:44.066992 17387 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:44.101847 13701 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:44.105196 13661 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:44.082142 17603 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.107920 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:44.123267 13705 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:44.126046 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1501) took 1391 ms. Trace:
I20260807 08:01:44.114771 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:44.068280 17703 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260807 08:01:43.988145 17654 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.121447 13661 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:44.133212 13661 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:44.129134 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:44.102818 17605 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:44.134460 14626 raft_consensus.cc:2362] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4. Re-sending same reply.
I20260807 08:01:44.107275 17697 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:44.136389 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:44.118726 17478 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:44.166330 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:44.165784 14629 raft_consensus.cc:2362] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5. Re-sending same reply.
I20260807 08:01:44.129886 13702 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:44.095597 17602 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.118340 17683 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:44.181689 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 101665us
I20260807 08:01:44.183121 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:44.130625 17714 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 08:01:44.199673 17655 consensus_meta.cc:287] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.704s user 0.010s sys 0.002s
I20260807 08:01:44.200915 17616 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.188066 13769 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:44.188956 17141 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:44.189473 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:44.135803 17645 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:44.171382 17609 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:44.100885 17566 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.224200 17645 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.225509 17696 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:44.227247 17566 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.202219 17617 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.197965 17683 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:44.203785 13703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1480) took 4385 ms (client timeout 2386 ms). Trace:
I20260807 08:01:44.204811 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:44.186578 13705 raft_consensus.cc:1226] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:44.200500 17655 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.114607 17429 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.113687 17427 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.206532 17439 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:44.202297 17596 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:44.211170 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.241557 17439 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:44.242321 14625 raft_consensus.cc:1226] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:44.243562 17655 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:44.244783 17602 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.224222 17609 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:44.210955 17536 consensus_queue.cc:1046] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
W20260807 08:01:44.212903 14622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 528) took 4241 ms (client timeout 3607 ms). Trace:
I20260807 08:01:44.192844 17603 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 08:01:44.247762 14622 rpcz_store.cc:269] 0807 08:01:39.971565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.971735 (+ 170us) service_pool.cc:227] Handling call
0807 08:01:44.212711 (+4240976us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b540064e6d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:44.212834 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1997501184}
I20260807 08:01:44.256024 17603 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:44.256619 17603 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:44.255563 14620 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:44.229460 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:44.229805 13703 rpcz_store.cc:269] 0807 08:01:39.818405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.818590 (+ 185us) service_pool.cc:227] Handling call
0807 08:01:44.203450 (+4384860us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54002b9154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:44.203626 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3054837504}
I20260807 08:01:44.117952 17440 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.232622 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:44.171481 17712 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:44.265230 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:44.266048 13585 raft_consensus.cc:2362] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5. Re-sending same reply.
I20260807 08:01:44.236603 13701 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:44.199255 17728 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:44.235109 17683 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.247244 17609 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:44.251061 17429 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.279605 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:44.279922 13537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 4.402s user 0.015s sys 0.038s
I20260807 08:01:44.252468 17536 consensus_queue.cc:1225] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer 9210dbc4cc9c45e5b26a463b705d362c log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260807 08:01:44.236968 17617 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:44.255342 17443 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.215232 17714 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:44.262981 13702 raft_consensus.cc:2379] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:44.264736 17498 raft_consensus.cc:3032] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:44.245230 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 540) took 1212 ms. Trace:
I20260807 08:01:44.272174 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:44.273203 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:44.274319 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:44.267122 17439 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:44.278666 14583 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:44.280748 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:44.234787 17630 raft_consensus.cc:686] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:44.279102 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:44.284672 14625 rpcz_store.cc:276] 0807 08:01:43.032950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.033094 (+ 144us) service_pool.cc:227] Handling call
0807 08:01:43.421513 (+388419us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:01:43.421778 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:44.242878 (+821100us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:44.244935 (+ 2057us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b54004deed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:44.245166 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1892857344}
I20260807 08:01:44.296197 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:44.296937 13703 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:44.297283 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 114048us
I20260807 08:01:44.285833 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:44.298964 13537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 4.427s user 0.015s sys 0.038s
I20260807 08:01:44.288959 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.278097 17367 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:44.299393 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 4.43563e+06us
I20260807 08:01:44.289047 13817 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:44.298512 13540 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:44.291194 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:44.282009 13769 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:44.284618 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:44.270448 17697 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.300791 13817 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:54072
I20260807 08:01:44.512725 17620 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.529090 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1733) took 1331 ms. Trace:
I20260807 08:01:44.588354 17581 raft_consensus.cc:3032] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Stepping down as leader of term 5
W20260807 08:01:44.607139 17681 consensus_meta.cc:287] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.686s user 0.013s sys 0.000s
I20260807 08:01:44.582454 17540 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:44.603546 17571 consensus_meta.cc:287] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.515s user 0.010s sys 0.003s
I20260807 08:01:44.126639 13710 rpcz_store.cc:276] 0807 08:01:42.734668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.196119 (+461451us) service_pool.cc:227] Handling call
0807 08:01:43.229977 (+ 33858us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:43.238234 (+ 8257us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:43.238266 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:43.239573 (+ 1307us) log.cc:842] Serialized 26 byte log entry
0807 08:01:43.383996 (+144423us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:44.017870 (+633874us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b180032c9e0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a025345 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a040155 00007f6b20940167 00007f6b1cf75807 00007f6b1cf84f0f 00007f6b1cf84e88
0807 08:01:44.115382 (+ 97512us) log.cc:842] Serialized 14 byte log entry
0807 08:01:44.117062 (+ 1680us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:44.117139 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:44.117178 (+ 39us) raft_consensus.cc:1671] finished
0807 08:01:44.125401 (+ 8223us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:44.125593 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7964,"prepare.queue_time_us":54380,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":8973,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":32567,"spinlock_wait_cycles":6242688,"thread_start_us":196349,"threads_started":2,"wal-append.queue_time_us":161384}
I20260807 08:01:44.358094 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:44.590387 13711 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260807 08:01:44.413626 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 135071us
W20260807 08:01:44.285472 13661 connection.cc:232] RPC call timeout handler was delayed by -1.0476152051705867s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3607s for a response
I20260807 08:01:44.423858 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.369766 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:44.607438 13814 raft_consensus.cc:1226] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
I20260807 08:01:44.606902 13587 rpcz_store.cc:276] 0807 08:01:43.197832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.197960 (+ 128us) service_pool.cc:227] Handling call
0807 08:01:43.464599 (+266639us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:43.511928 (+ 47329us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:44.528872 (+1016944us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54001e3f50. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:44.529025 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":537565952}
I20260807 08:01:44.306949 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:44.626488 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:44.291967 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:44.368316 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:44.633464 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.605832 17550 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:44.649334 17541 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:44.753001 17508 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:44.753479 17541 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:44.754808 17218 consensus_meta.cc:287] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.648s user 0.013s sys 0.000s
I20260807 08:01:44.755373 17218 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.744690 14625 raft_consensus.cc:2379] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:44.390062 13585 raft_consensus.cc:1226] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260807 08:01:44.745108 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1753) took 1738 ms. Trace:
I20260807 08:01:44.760111 13711 rpcz_store.cc:276] 0807 08:01:43.006460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.875813 (+869353us) service_pool.cc:227] Handling call
0807 08:01:43.916335 (+ 40522us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:44.590047 (+673712us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54000fe4a0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cefb730 00007f6b1cf6d300 00007f6b1cf6e371 00007f6b1cf6aedf 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc
0807 08:01:44.744845 (+154798us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:44.745055 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":466062208}
I20260807 08:01:44.387650 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:44.413228 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.655035 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1802) took 1766 ms. Trace:
I20260807 08:01:44.763648 13590 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:44.516582 13703 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:44.765053 13703 raft_consensus.cc:1226] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 3. Current term is 5. Ops: []
I20260807 08:01:44.747376 14626 raft_consensus.cc:1226] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260807 08:01:44.748385 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:44.630394 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 348488us
W20260807 08:01:44.743263 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 436363us
I20260807 08:01:44.331687 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:44.770810 13769 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:44.764083 13591 rpcz_store.cc:276] 0807 08:01:42.888837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:42.998228 (+109391us) service_pool.cc:227] Handling call
0807 08:01:44.654765 (+1656537us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54001644d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:44.654973 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":341741824}
I20260807 08:01:44.771765 17735 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:44.762897 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:44.772890 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:44.773123 13587 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 6
I20260807 08:01:44.773538 14629 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:44.288664 17714 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:44.496788 17721 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:44.607399 17581 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:44.537807 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:44.601687 17440 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:44.329921 17537 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:44.288683 17498 raft_consensus.cc:729] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
W20260807 08:01:44.765784 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 141921us
I20260807 08:01:44.767318 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.564308 17741 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:44.289124 17429 consensus_queue.cc:1046] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.488s
I20260807 08:01:44.525484 16893 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:44.590445 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:44.766862 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.622593 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:44.771729 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:44.776190 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.498164 17639 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:44.575564 14618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 523) took 4773 ms (client timeout 2131 ms). Trace:
I20260807 08:01:44.429962 17683 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:44.600754 17548 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:44.460498 17723 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:44.410228 17654 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:44.779597 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 531) took 3683 ms. Trace:
I20260807 08:01:44.779462 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.628176 17697 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:44.396219 17645 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.630084 17489 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:44.291369 17617 raft_consensus.cc:686] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:44.386488 17734 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.550s
W20260807 08:01:44.818145 14618 rpcz_store.cc:269] 0807 08:01:39.802007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.805820 (+ 3813us) service_pool.cc:227] Handling call
0807 08:01:44.575198 (+4769378us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54006a04d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:44.575450 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2979723648}
I20260807 08:01:44.352705 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:44.780766 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:44.782219 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.782311 13769 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:44.724438 17480 consensus_queue.cc:1046] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260807 08:01:44.774685 13661 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:44.448441 13700 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 385) took 4774 ms (client timeout 2062 ms). Trace:
W20260807 08:01:44.839504 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 57290.3us
I20260807 08:01:44.795171 13579 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 5
I20260807 08:01:44.591924 17387 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:44.798733 14623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 543) took 1765 ms (client timeout 1799 ms). Trace:
I20260807 08:01:44.781751 17537 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:44.801801 16893 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.803526 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:44.750770 17569 consensus_queue.cc:1046] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260807 08:01:44.798367 13540 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:44.777755 17616 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:44.605506 17609 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.820737 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.626662 17605 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.757755 17571 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.825708 13814 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:44.881006 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.825441 14630 rpcz_store.cc:276] 0807 08:01:41.095910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.180041 (+ 84131us) service_pool.cc:227] Handling call
0807 08:01:42.247960 (+1067919us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:42.429483 (+181523us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:42.902945 (+473462us) spinlock_profiling.cc:243] Waited 68.2 ms on lock 0x7b540043c6d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:44.779399 (+1876454us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b540043c6d0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:44.779539 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2410058368}
I20260807 08:01:44.782414 13591 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260807 08:01:44.831133 13805 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:44.607643 17681 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.304493 17652 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.293422 17439 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:44.886059 17541 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.886166 17439 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:44.886940 17739 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:44.887434 17739 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.806249 17508 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:44.887902 17652 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:44.840932 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 66343.3us
I20260807 08:01:44.889482 13711 raft_consensus.cc:1226] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:44.792806 17721 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.890820 13702 raft_consensus.cc:1226] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:44.891040 17681 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:44.850243 14623 rpcz_store.cc:269] 0807 08:01:43.033607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.034238 (+ 631us) service_pool.cc:227] Handling call
0807 08:01:44.533023 (+1498785us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:44.798646 (+265623us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9423744}
I20260807 08:01:44.892539 17721 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:44.827409 17617 consensus_queue.cc:237] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.842064 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.895607 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.897022 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.844337 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
W20260807 08:01:44.898208 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 53942.6us
I20260807 08:01:44.899353 17571 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.899844 13769 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:44.855558 17537 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:44.850715 17697 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:44.828007 17440 raft_consensus.cc:3032] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:44.844983 17489 raft_consensus.cc:686] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:01:44.839525 13700 rpcz_store.cc:269] 0807 08:01:39.674277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:41.169768 (+1495491us) service_pool.cc:227] Handling call
0807 08:01:44.440746 (+3270978us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b54003a98d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:44.440996 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1037511936}
I20260807 08:01:44.842890 14620 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:44.849606 17707 consensus_queue.cc:1046] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20260807 08:01:44.912038 13590 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:44.288674 17728 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.852295 13589 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:44.853907 14629 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:44.839663 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 70569.1us
I20260807 08:01:44.865321 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.914255 13817 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:44.821301 17639 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.918678 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.919500 13805 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:44.914582 13807 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:44.883925 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:44.889321 13661 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:44.419992 17630 consensus_queue.cc:237] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.894007 17387 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.901470 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:44.883229 17522 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:44.850304 17645 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:44.914736 13817 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:44.926185 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:44.900991 17548 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:44.850373 17734 consensus_queue.cc:1046] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260807 08:01:44.857908 17218 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:44.814520 17741 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.897315 16893 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:44.904757 17440 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:01:45.080547 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 141551us
I20260807 08:01:44.841238 17581 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:44.879561 17616 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:44.939460 17697 raft_consensus.cc:686] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:45.080454 17712 consensus_meta.cc:287] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.815s user 0.005s sys 0.010s
I20260807 08:01:45.080085 17659 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:45.081684 17548 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:45.087409 17522 raft_consensus.cc:686] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:45.092376 17739 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:45.099427 17645 raft_consensus.cc:686] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:45.123060 17712 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:45.132251 17659 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.158516 17605 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:45.138233 17548 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:45.159550 17609 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.160447 17489 consensus_queue.cc:237] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:45.120080 17679 consensus_meta.cc:287] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.193s user 0.000s sys 0.014s
I20260807 08:01:45.166491 17697 consensus_queue.cc:237] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:45.165483 17603 consensus_meta.cc:287] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.908s user 0.008s sys 0.004s
I20260807 08:01:45.182772 17522 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.152736 17728 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:45.029474 17683 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:45.151409 17639 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:44.939137 17735 raft_consensus.cc:686] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:45.170158 17581 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:45.176013 17616 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:45.169570 17741 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:45.265403 17683 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:45.257057 17597 raft_consensus.cc:3032] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:45.170960 17645 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.273577 17603 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.276893 17659 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:45.198504 17739 consensus_queue.cc:1046] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.108s
I20260807 08:01:45.305721 17723 raft_consensus.cc:3032] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:45.230147 17679 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.299643 17683 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:01:45.311591 17749 log.cc:924] Time spent T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.281s user 0.004s sys 0.000s
I20260807 08:01:45.313341 17723 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:45.313215 17735 consensus_queue.cc:237] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:44.878214 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 79935.7us
I20260807 08:01:45.354583 13540 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:45.356573 13540 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:45.356618 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:45.358325 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:44.911831 13587 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:45.360878 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:44.913295 13579 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:45.362951 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1823) took 1032 ms. Trace:
I20260807 08:01:45.363397 13579 rpcz_store.cc:276] 0807 08:01:44.330494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.331530 (+ 1036us) service_pool.cc:227] Handling call
0807 08:01:45.362718 (+1031188us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:45.386272 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 496879us
I20260807 08:01:45.386706 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 399) took 1152 ms. Trace:
I20260807 08:01:45.387169 13705 rpcz_store.cc:276] 0807 08:01:44.234313 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.744766 (+510453us) service_pool.cc:227] Handling call
0807 08:01:44.745120 (+ 354us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:44.761485 (+ 16365us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:44.761538 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:44.778045 (+ 16507us) log.cc:842] Serialized 52 byte log entry
0807 08:01:44.844861 (+ 66816us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:44.844905 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:44.847018 (+ 2113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:45.310936 (+463918us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:45.385814 (+ 74878us) raft_consensus.cc:1671] finished
0807 08:01:45.385869 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:45.386436 (+ 567us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18357,"raft.queue_time_us":20176,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"spinlock_wait_cycles":7076096,"thread_start_us":39559,"threads_started":1,"wal-append.queue_time_us":103575}
I20260807 08:01:45.394044 13661 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.396682 13661 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:44.944218 13769 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:45.405237 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 460924us
I20260807 08:01:45.412325 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.404016 13661 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.418102 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.419293 13805 raft_consensus.cc:1226] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:45.420269 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1513) took 1003 ms. Trace:
I20260807 08:01:45.420614 13805 rpcz_store.cc:276] 0807 08:01:44.417075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.942482 (+525407us) service_pool.cc:227] Handling call
0807 08:01:45.418609 (+476127us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:45.419950 (+ 1341us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.420177 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:45.421650 13700 raft_consensus.cc:1226] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:45.422391 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:45.423185 13805 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:45.424597 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.427915 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:45.419296 13705 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:45.428797 13700 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:45.419232 13661 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.432684 13661 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.436764 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:45.437615 13811 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:45.438549 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.438798 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:45.439720 13705 raft_consensus.cc:2396] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260807 08:01:45.440012 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:45.440755 13811 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:45.443331 13705 raft_consensus.cc:1226] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260807 08:01:45.446182 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:45.455070 17368 raft_consensus.cc:2781] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:45.456013 17368 raft_consensus.cc:686] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:45.169361 17439 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.460438 13811 raft_consensus.cc:1261] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260807 08:01:45.460892 13811 pending_rounds.cc:76] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 3
I20260807 08:01:45.461194 13811 pending_rounds.cc:97] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260807 08:01:45.167039 17440 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.464540 17486 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:45.467706 17524 consensus_queue.cc:1046] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:45.276337 17552 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.470753 13808 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:45.474131 17368 consensus_queue.cc:237] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:45.475026 17550 consensus_meta.cc:287] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.719s user 0.013s sys 0.000s
I20260807 08:01:45.516032 13812 raft_consensus.cc:1261] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260807 08:01:45.516396 13812 pending_rounds.cc:76] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 6
I20260807 08:01:45.516621 13812 pending_rounds.cc:97] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260807 08:01:45.516844 13812 raft_consensus.cc:2866] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:45.517778 17736 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:45.475679 17439 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.469609 17569 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:45.518414 17736 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.479426 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:45.469710 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:45.519929 14623 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:45.481544 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:45.520488 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:45.481994 16459 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:45.521790 16459 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.526957 17679 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.482306 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.299283 17597 raft_consensus.cc:729] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:45.528016 13710 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:45.529060 17597 consensus_queue.cc:260] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:45.531208 17597 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:45.535025 17440 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:45.535398 13592 raft_consensus.cc:3032] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:45.535883 13592 raft_consensus.cc:729] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:45.313215 17714 consensus_meta.cc:287] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.531s user 0.007s sys 0.003s
I20260807 08:01:45.503651 13817 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:45.504513 17739 consensus_queue.cc:1225] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer 77da774f2bb54cf28302d53d0db224ad log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260807 08:01:45.506039 17609 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:45.562461 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 402) took 1324 ms. Trace:
I20260807 08:01:45.506464 17593 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:45.506944 17734 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:45.507356 17751 consensus_queue.cc:1046] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260807 08:01:45.570070 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:45.562847 13711 rpcz_store.cc:276] 0807 08:01:44.237984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.761454 (+523470us) service_pool.cc:227] Handling call
0807 08:01:44.778708 (+ 17254us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:44.890078 (+111370us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.561945 (+671867us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54002b9b54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:45.562264 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":254245760}
I20260807 08:01:45.513458 17480 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 08:01:45.572345 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 139951us
W20260807 08:01:45.544442 13709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1761) took 2368 ms (client timeout 2080 ms). Trace:
I20260807 08:01:45.573063 17734 raft_consensus.cc:686] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:45.510134 17652 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:45.511015 17654 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:45.511827 17681 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:45.573895 13661 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:45.575309 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:45.576561 13661 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:45.510013 17723 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.512354 13701 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:45.476209 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:45.582687 17654 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:45.583523 17654 raft_consensus.cc:686] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:45.496601 13811 raft_consensus.cc:2866] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:45.559934 17569 raft_consensus.cc:686] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:45.588699 13700 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:45.570216 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 394) took 1337 ms. Trace:
I20260807 08:01:45.543349 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1493) took 2869 ms. Trace:
I20260807 08:01:45.540474 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1827) took 1147 ms. Trace:
I20260807 08:01:45.568917 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1863) took 1581 ms. Trace:
I20260807 08:01:45.542065 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1752) took 2636 ms. Trace:
I20260807 08:01:45.584156 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 400) took 1344 ms. Trace:
W20260807 08:01:45.542317 17537 consensus_meta.cc:287] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.641s user 0.011s sys 0.003s
I20260807 08:01:45.575327 17681 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:45.597416 13814 raft_consensus.cc:1226] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 8. Ops: []
W20260807 08:01:45.573063 13709 rpcz_store.cc:269] 0807 08:01:43.175464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.178142 (+ 2678us) service_pool.cc:227] Handling call
0807 08:01:45.544002 (+2365860us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54001dc4d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:45.544210 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":242993024}
I20260807 08:01:45.589491 13708 rpcz_store.cc:276] 0807 08:01:42.674205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:42.684308 (+ 10103us) service_pool.cc:227] Handling call
0807 08:01:42.702797 (+ 18489us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:43.200406 (+497609us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.542828 (+2342422us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b5400081bd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:45.543194 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1109403008}
I20260807 08:01:45.589954 13809 rpcz_store.cc:276] 0807 08:01:43.981480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.982021 (+ 541us) service_pool.cc:227] Handling call
0807 08:01:43.982409 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:44.092497 (+110088us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:44.092542 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:44.092595 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:44.092639 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.562270 (+1469631us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400125f54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:45.562330 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:45.562568 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":252634880}
I20260807 08:01:45.591951 13701 rpcz_store.cc:276] 0807 08:01:44.234592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.313724 (+ 79132us) service_pool.cc:227] Handling call
0807 08:01:45.578472 (+1264748us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b5400082fd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:45.578650 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1703757696}
I20260807 08:01:45.589432 13702 rpcz_store.cc:276] 0807 08:01:44.232643 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.370171 (+137528us) service_pool.cc:227] Handling call
0807 08:01:44.370565 (+ 394us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:44.891348 (+520783us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.569674 (+678326us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b540039d354. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:45.570025 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1195933568}
I20260807 08:01:45.589896 13589 rpcz_store.cc:276] 0807 08:01:44.393293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.626159 (+232866us) service_pool.cc:227] Handling call
0807 08:01:45.540069 (+913910us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54001e3054. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:45.540297 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":248762112}
I20260807 08:01:45.597976 13705 raft_consensus.cc:2379] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:45.599228 13805 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:45.544103 17524 raft_consensus.cc:3032] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:45.537150 13592 consensus_queue.cc:260] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:45.608449 13418 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 225671us
I20260807 08:01:45.520504 17732 consensus_queue.cc:1046] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.105s
I20260807 08:01:45.582752 17714 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.609583 17734 consensus_queue.cc:237] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:45.542711 17486 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:45.591423 13706 rpcz_store.cc:276] 0807 08:01:42.905798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.197302 (+291504us) service_pool.cc:227] Handling call
0807 08:01:43.980249 (+782947us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:44.012719 (+ 32470us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.541663 (+1528944us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b54002b9650. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:45.541850 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1799423232}
I20260807 08:01:45.628225 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:45.611029 13588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56576 (request call id 971) took 1716 ms. Trace:
I20260807 08:01:45.611881 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1516) took 1193 ms. Trace:
I20260807 08:01:45.612107 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:45.614470 17728 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:45.621690 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 45319.7us
I20260807 08:01:45.610917 17603 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.641559 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1525) took 1011 ms. Trace:
I20260807 08:01:45.656196 13588 rpcz_store.cc:276] 0807 08:01:43.815855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.816033 (+ 178us) service_pool.cc:227] Handling call
0807 08:01:45.530294 (+1714261us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54001e4e54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0af0 00007f6b1cf73f2a 00007f6b21eb0c6e 00007f6b1b5a7807 00007f6b1b5a7792 00007f6b1b5a76c1 00007f6b1b5a763c 00007f6b1b5a68f2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:45.532249 (+ 1955us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":301317888}
I20260807 08:01:45.655745 13814 rpcz_store.cc:276] 0807 08:01:44.418529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.942618 (+524089us) service_pool.cc:227] Handling call
0807 08:01:44.943025 (+ 407us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:45.610225 (+667200us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.611495 (+ 1270us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b54001b7454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:45.611789 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1476323712}
I20260807 08:01:45.679100 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:45.622026 17593 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:45.659198 13808 rpcz_store.cc:276] 0807 08:01:44.630512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:45.433861 (+803349us) service_pool.cc:227] Handling call
0807 08:01:45.434411 (+ 550us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:45.514188 (+ 79777us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.637777 (+123589us) spinlock_profiling.cc:243] Waited 35.9 ms on lock 0x7b540037afd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:45.641469 (+ 3692us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82483584}
I20260807 08:01:45.637979 17524 raft_consensus.cc:729] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:45.618158 16459 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:45.628011 17578 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:45.627230 17427 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:45.622190 17681 raft_consensus.cc:686] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:45.699746 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:45.702421 14619 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:45.709149 13661 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.710501 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1871) took 1432 ms. Trace:
I20260807 08:01:45.711030 13661 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.681248 17728 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.622910 17498 consensus_queue.cc:260] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.719614 17498 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:45.720345 14629 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260807 08:01:45.722021 13809 raft_consensus.cc:1261] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 08:01:45.615567 17537 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.722851 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:45.723513 13809 raft_consensus.cc:1226] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:45.723874 13706 raft_consensus.cc:2379] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 7: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:45.728004 17524 consensus_queue.cc:260] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.629560 17654 consensus_queue.cc:237] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:45.731642 13769 connection.cc:232] RPC call timeout handler was delayed by -1.3187508322298527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20260807 08:01:45.732089 13769 connection.cc:232] RPC call timeout handler was delayed by -1.305250832112506s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15760000000000002s for a response
W20260807 08:01:45.732366 13769 connection.cc:232] RPC call timeout handler was delayed by -1.2242508321069181s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
W20260807 08:01:45.732722 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1171508322004229s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
W20260807 08:01:45.733067 13769 connection.cc:232] RPC call timeout handler was delayed by -1.0145508321002126s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
I20260807 08:01:45.736068 13814 raft_consensus.cc:1226] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260807 08:01:45.740096 17723 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:45.740195 13814 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:45.710908 13816 rpcz_store.cc:276] 0807 08:01:44.277671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.277855 (+ 184us) service_pool.cc:227] Handling call
0807 08:01:45.710096 (+1432241us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001fd1d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:45.710288 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2819753856}
I20260807 08:01:45.623802 17548 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.749408 17427 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:45.757925 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.760161 13706 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:45.760087 17427 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.757678 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:45.634666 17569 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.751595 17524 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:45.767150 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:45.769026 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 180721us
I20260807 08:01:45.750989 17593 raft_consensus.cc:686] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:45.763314 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:45.775071 13700 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
W20260807 08:01:45.765652 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 409289us
I20260807 08:01:45.754155 17652 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:45.752216 17537 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:45.757598 17609 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:45.779353 13808 raft_consensus.cc:1226] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 5. Current term is 6. Ops: []
W20260807 08:01:45.780642 13418 catalog_manager.cc:4584] TS 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): ChangeConfig:REMOVE_PEER RPC failed for tablet 0248ade936924e628c46b8de6252af1b: Remote error: Service unavailable: leader transfer in progress
I20260807 08:01:45.780843 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:45.781867 14618 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:45.760187 17603 consensus_queue.cc:1046] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260807 08:01:45.784111 17603 raft_consensus.cc:3032] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:45.784174 17585 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:45.784636 17603 raft_consensus.cc:729] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:45.790199 13816 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260807 08:01:45.790541 13816 pending_rounds.cc:76] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 3
I20260807 08:01:45.790691 13816 pending_rounds.cc:97] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260807 08:01:45.790835 13816 raft_consensus.cc:2866] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:45.793753 17499 raft_consensus.cc:3032] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:01:45.786815 13661 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.817397 17499 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:45.818368 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 62618.2us
I20260807 08:01:45.818526 17499 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:45.819597 17499 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:45.821806 17576 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:45.795514 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:45.798251 17548 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.797806 17603 consensus_queue.cc:260] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.798918 17486 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:45.800473 17427 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:45.801013 17581 consensus_meta.cc:287] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.507s user 0.000s sys 0.014s
I20260807 08:01:45.801409 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.801759 13706 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260807 08:01:45.802922 17714 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.803366 13537 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.805799 17683 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:45.807179 13587 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260807 08:01:45.835634 17489 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:45.836285 17489 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.837782 13657 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:45.839018 17683 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:45.839355 13657 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:45.842046 17569 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:45.842859 13700 raft_consensus.cc:2379] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 7: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:45.830896 17728 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.808182 17593 consensus_queue.cc:237] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.809845 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:45.812187 13805 raft_consensus.cc:2396] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:01:45.814113 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.814819 13701 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:45.815392 14620 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:45.845821 13817 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:45.846421 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:54072 (request call id 1509) took 1545 ms. Trace:
I20260807 08:01:45.846565 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.846787 13817 rpcz_store.cc:276] 0807 08:01:44.300575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.300682 (+ 107us) service_pool.cc:227] Handling call
0807 08:01:45.844752 (+1544070us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54001b7454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf62da7 00007f6b21eaeee0 00007f6b1b5a0784 00007f6b1b5a0712 00007f6b1b5a0641 00007f6b1b5a05bc 00007f6b1b59f872 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:45.846278 (+ 1526us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":709608832}
I20260807 08:01:45.809990 13709 raft_consensus.cc:1226] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260807 08:01:45.815877 17597 consensus_queue.cc:1046] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:45.815434 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.824744 13814 raft_consensus.cc:1226] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:45.816680 14623 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:45.842691 13537 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.815601 14618 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:45.830535 13579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0248ade936924e628c46b8de6252af1b"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56576
I20260807 08:01:45.859103 13579 raft_consensus.cc:595] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Received request to transfer leadership
I20260807 08:01:45.859338 14620 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:45.860577 13702 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:45.860854 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.861840 13711 raft_consensus.cc:2396] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260807 08:01:45.851645 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:45.865748 13809 raft_consensus.cc:1226] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:45.863421 17736 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.870678 13706 raft_consensus.cc:1226] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:45.872174 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.854705 17486 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:45.870692 17569 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:45.875047 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.876008 13700 raft_consensus.cc:1226] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:45.875188 13710 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260807 08:01:45.877693 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:45.873322 13579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0248ade936924e628c46b8de6252af1b"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56576
I20260807 08:01:45.885694 13579 raft_consensus.cc:595] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Received request to transfer leadership
W20260807 08:01:45.889055 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 46491.2us
I20260807 08:01:45.889868 14618 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260807 08:01:45.889810 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:45.877364 17486 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:45.878048 17607 raft_consensus.cc:3032] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:45.877885 17576 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:45.877187 17603 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:45.900642 13537 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:45.903559 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 75052.9us
W20260807 08:01:45.901554 13508 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 0248ade936924e628c46b8de6252af1b already in progress
I20260807 08:01:45.900700 13584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:39822 (request call id 18) took 3222 ms. Trace:
I20260807 08:01:45.904685 14629 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:45.901768 13804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 540) took 2082 ms. Trace:
I20260807 08:01:45.911604 17571 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:45.897964 17607 raft_consensus.cc:729] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:45.911808 13661 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:45.912753 14580 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:45.904075 13508 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 0248ade936924e628c46b8de6252af1b already in progress
I20260807 08:01:45.913439 17714 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:45.913067 17607 consensus_queue.cc:260] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:45.913873 17571 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:45.914357 17571 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:45.915444 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:45.904382 17576 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:45.916873 17576 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:45.897760 17486 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:45.917742 17681 consensus_queue.cc:237] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.904295 13584 rpcz_store.cc:276] 0807 08:01:42.678024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:42.678198 (+ 174us) service_pool.cc:227] Handling call
0807 08:01:45.892957 (+3214759us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b54002ac454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf65bc3 00007f6b21eaf81a 00007f6b1b5a2d04 00007f6b1b5a2c92 00007f6b1b5a2bc1 00007f6b1b5a2b3c 00007f6b1b5a1df2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:45.900493 (+ 7536us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":794197376}
I20260807 08:01:45.923810 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:45.924149 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 94245.8us
I20260807 08:01:45.925953 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:45.926645 13705 raft_consensus.cc:1226] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
I20260807 08:01:45.927964 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:45.928563 17429 raft_consensus.cc:3032] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:45.928833 13805 raft_consensus.cc:2379] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 4: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:45.929001 17429 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:45.929943 14580 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.930542 17540 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:45.930150 17429 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:45.916893 17489 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:45.931370 17429 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:45.931864 17655 consensus_queue.cc:1046] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:45.932252 17537 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:45.933652 17537 consensus_queue.cc:1046] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:01:45.906567 13804 rpcz_store.cc:276] 0807 08:01:43.819785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.819979 (+ 194us) service_pool.cc:227] Handling call
0807 08:01:43.820366 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:43.820618 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 08:01:43.976179 (+155561us) log.cc:842] Serialized 14 byte log entry
0807 08:01:44.094266 (+118087us) log.cc:842] Serialized 14 byte log entry
0807 08:01:45.891958 (+1797692us) spinlock_profiling.cc:243] Waited 9.53 ms on lock 0x7b180035e1e0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1cbfa82c 00007f6b1a0259dc 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a040155 00007f6b1cf633d0 00007f6b1cf75576 00007f6b1cf860d7
0807 08:01:45.901056 (+ 9098us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:45.901127 (+ 71us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:45.901269 (+ 142us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.901361 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:45.901580 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":408091,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":73,"raft.queue_time_us":79073,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":159,"spinlock_wait_cycles":21921024,"thread_start_us":343657,"threads_started":3,"wal-append.queue_time_us":137776,"wal-append.run_cpu_time_us":2386,"wal-append.run_wall_time_us":1113549}
I20260807 08:01:45.934469 17655 raft_consensus.cc:3032] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:45.934926 17655 raft_consensus.cc:729] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:45.935101 17696 raft_consensus.cc:3032] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:45.935456 17722 raft_consensus.cc:3032] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:45.935559 17696 raft_consensus.cc:729] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:45.931033 17643 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:45.927081 13817 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
W20260807 08:01:45.936713 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 46917.8us
I20260807 08:01:45.940842 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260807 08:01:45.943190 17696 consensus_queue.cc:260] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.945114 17696 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:45.945814 14627 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:45.946107 13592 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:45.946579 17449 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:45.947431 14627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 541) took 2914 ms (client timeout 1987 ms). Trace:
I20260807 08:01:45.947077 17449 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:45.947901 14627 rpcz_store.cc:269] 0807 08:01:43.033184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.033310 (+ 126us) service_pool.cc:227] Handling call
0807 08:01:45.947176 (+2913866us) spinlock_profiling.cc:243] Waited 2.8 s on lock 0x7b54004b32d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:45.947331 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6451373824}
I20260807 08:01:45.948890 13812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1843) took 3040 ms. Trace:
I20260807 08:01:45.947085 14619 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:45.950188 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 49680.5us
I20260807 08:01:45.951122 13537 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:45.938673 17714 raft_consensus.cc:686] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:45.937929 17368 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:45.951864 17449 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:45.950292 14619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 548) took 1670 ms. Trace:
I20260807 08:01:45.952188 17368 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:45.952379 17714 consensus_queue.cc:237] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.952795 17508 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:45.952973 17368 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:45.954172 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 48297.4us
I20260807 08:01:45.954864 13657 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:45.952903 13537 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:45.957625 13814 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:45.952611 14619 rpcz_store.cc:276] 0807 08:01:44.280234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.282205 (+ 1971us) service_pool.cc:227] Handling call
0807 08:01:45.611608 (+1329403us) spinlock_profiling.cc:243] Waited 5.81 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:01:45.950217 (+338609us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13373184}
I20260807 08:01:45.959527 13537 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:45.934573 17537 raft_consensus.cc:3032] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:45.949278 13812 rpcz_store.cc:276] 0807 08:01:42.886074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:42.886631 (+ 557us) service_pool.cc:227] Handling call
0807 08:01:42.945290 (+ 58659us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:45.517410 (+2572120us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.902138 (+384728us) spinlock_profiling.cc:243] Waited 2.57 s on lock 0x7b540006bb54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:45.926582 (+ 24444us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5910694656}
I20260807 08:01:45.967540 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:45.969187 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 65045.2us
I20260807 08:01:45.969419 13814 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:45.971580 17541 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:45.971971 17541 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:45.972352 17541 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:45.975291 13657 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:45.975489 13592 raft_consensus.cc:1261] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260807 08:01:45.975896 13592 pending_rounds.cc:76] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Aborting all ops after (but not including) 2
I20260807 08:01:45.976142 13592 pending_rounds.cc:97] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260807 08:01:45.976384 13592 raft_consensus.cc:2866] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:45.976563 17581 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:45.981693 17581 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:45.977058 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1819) took 1702 ms. Trace:
W20260807 08:01:45.977182 17658 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260807 08:01:45.935920 17722 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:45.966804 17508 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:45.979732 13817 raft_consensus.cc:3032] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:45.979965 17641 consensus_queue.cc:1035] T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:45.986701 13592 rpcz_store.cc:276] 0807 08:01:44.275154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.275264 (+ 110us) service_pool.cc:227] Handling call
0807 08:01:44.275589 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:45.535123 (+1259534us) spinlock_profiling.cc:243] Waited 7.68 ms on lock 0x7b54003825a0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cefb730 00007f6b1cf6d300 00007f6b1cf6e371 00007f6b1cf6aedf 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc
0807 08:01:45.945945 (+410822us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b58007724b0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cee6f63 00007f6b1cf4aaba 00007f6b1cf623af 00007f6b1cf63b4e 00007f6b1cf6dc89 00007f6b1cf6e556 00007f6b1cf6aedf 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684
0807 08:01:45.976790 (+ 30845us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.977006 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":484383872}
I20260807 08:01:45.980369 13537 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:45.971580 17190 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260807 08:01:45.987883 13817 raft_consensus.cc:729] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:45.968384 17537 raft_consensus.cc:729] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:46.009600 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:46.010242 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.014968 13817 consensus_queue.cc:260] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:46.017894 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.018735 14622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 552) took 1634 ms. Trace:
I20260807 08:01:45.989308 17609 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:46.021474 13537 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:46.022604 17508 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:46.023367 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 532) took 1255 ms. Trace:
I20260807 08:01:46.034821 14583 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:46.027227 14622 rpcz_store.cc:276] 0807 08:01:44.353458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.353700 (+ 242us) service_pool.cc:227] Handling call
0807 08:01:44.354283 (+ 583us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:44.354693 (+ 410us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:44.354731 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:44.387526 (+ 32795us) log.cc:842] Serialized 26 byte log entry
0807 08:01:45.737569 (+1350043us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b180004d7a0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a025345 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a042e22 00007f6b1cbec5bb 00007f6b1cbf2063 00007f6b1cbf2411 00007f6b1cf3f5cf
0807 08:01:45.824769 (+ 87200us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:45.828308 (+ 3539us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:45.834964 (+ 6656us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:45.984179 (+149215us) raft_consensus.cc:1671] finished
0807 08:01:45.984196 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:45.987483 (+ 3287us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":39775,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":12720,"spinlock_wait_cycles":35704064,"thread_start_us":37149,"threads_started":1,"wal-append.queue_time_us":1480145}
I20260807 08:01:46.044131 17697 consensus_queue.cc:1035] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260807 08:01:46.047216 13537 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:46.048494 13537 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:46.049829 13817 raft_consensus.cc:3037] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:46.051101 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:46.035005 17508 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:46.052054 17508 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:46.052541 17722 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.053862 13584 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:46.054214 17783 log.cc:924] Time spent T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260807 08:01:46.055482 17647 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:46.056548 17607 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:46.056316 17537 consensus_queue.cc:260] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.057397 13591 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:46.057541 17537 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:46.059775 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:46.063982 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:46.065958 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:46.056903 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:46.070542 17498 raft_consensus.cc:2726] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:45.961537 17655 consensus_queue.cc:260] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.075951 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:46.041706 17581 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:46.035456 13580 rpcz_store.cc:276] 0807 08:01:44.767841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.802787 (+ 34946us) service_pool.cc:227] Handling call
0807 08:01:44.803104 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:44.902135 (+ 99031us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:44.902183 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:44.909794 (+ 7611us) log.cc:842] Serialized 26 byte log entry
0807 08:01:45.527755 (+617961us) spinlock_profiling.cc:243] Waited 5.34 ms on lock 0x7b180042f940. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a025345 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a042e22 00007f6b1cbec5bb 00007f6b1cbf2063 00007f6b1cbf2411 00007f6b1cf3f5cf
0807 08:01:45.751130 (+223375us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:45.751179 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:46.009770 (+258591us) spinlock_profiling.cc:243] Waited 98.6 ms on lock 0x7b540025ec54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:46.009830 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:46.009877 (+ 47us) raft_consensus.cc:1671] finished
0807 08:01:46.009894 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:46.023192 (+ 13298us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3697,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":5608,"spinlock_wait_cycles":239064192,"thread_start_us":218802,"threads_started":1,"wal-append.queue_time_us":873448}
I20260807 08:01:46.083192 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.053879 17722 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:46.039855 17585 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:46.086591 13817 raft_consensus.cc:1261] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260807 08:01:46.069336 17609 raft_consensus.cc:686] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:46.090576 13817 pending_rounds.cc:76] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 2
I20260807 08:01:46.093320 13817 pending_rounds.cc:97] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 08:01:46.093883 13817 raft_consensus.cc:2866] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:46.097913 17655 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
W20260807 08:01:46.098294 17785 log.cc:924] Time spent T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.162s user 0.003s sys 0.000s
I20260807 08:01:46.100591 14580 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:46.074535 17524 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:46.105496 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:46.107296 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 72563.3us
I20260807 08:01:46.082297 17654 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:46.087630 17658 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:46.129016 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.120522 13817 raft_consensus.cc:1226] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260807 08:01:46.106897 17524 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:46.145183 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:46.112396 17609 consensus_queue.cc:237] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.063431 17541 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.154930 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:46.154837 17368 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.134948 17449 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:46.161867 17617 consensus_queue.cc:1035] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
W20260807 08:01:46.058871 17643 consensus_queue.cc:1159] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "77da774f2bb54cf28302d53d0db224ad" responder_term: 7 status { last_received { term: 5 index: 7 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: [6.7-6.7]" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 08:01:46.178794 13703 raft_consensus.cc:1226] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260807 08:01:46.166934 17527 consensus_queue.cc:1046] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260807 08:01:46.167667 14627 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:46.180949 17527 raft_consensus.cc:3032] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:46.126359 17658 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:46.181331 17527 raft_consensus.cc:729] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:46.181594 17658 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:46.181783 17499 consensus_queue.cc:1046] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:46.182204 17527 consensus_queue.cc:260] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.183434 17499 raft_consensus.cc:3032] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:46.183964 17499 raft_consensus.cc:729] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:46.184474 13580 raft_consensus.cc:2379] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:46.153087 17524 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:46.186832 17541 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:46.188844 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:46.173635 13808 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260807 08:01:46.190016 13808 pending_rounds.cc:76] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 4
I20260807 08:01:46.190251 13808 pending_rounds.cc:97] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260807 08:01:46.190470 13808 raft_consensus.cc:2866] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:46.190526 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.175292 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:46.134428 17585 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.177896 17368 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:46.183514 17527 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:46.176285 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:46.217953 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.216905 13587 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:46.223174 17449 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:46.217629 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.217931 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.224014 13589 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:46.218291 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.223174 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.226022 13588 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:46.218396 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.226624 17522 consensus_queue.cc:1046] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:46.226895 13585 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:46.227558 17522 consensus_queue.cc:1046] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:46.228339 17696 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:46.228643 17765 consensus_queue.cc:1046] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:46.229547 17765 raft_consensus.cc:3032] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:46.229893 17765 raft_consensus.cc:729] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:46.230166 17696 raft_consensus.cc:3032] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:46.230564 17696 raft_consensus.cc:729] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:46.230732 17765 consensus_queue.cc:260] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:46.231577 17522 raft_consensus.cc:3032] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:46.231926 17522 raft_consensus.cc:729] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:46.232079 17765 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 8
I20260807 08:01:46.233923 17368 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:46.234504 17585 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:46.234988 17696 consensus_queue.cc:260] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:46.236819 17696 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:46.232815 17522 consensus_queue.cc:260] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.238449 17522 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:46.239228 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:46.240231 13804 raft_consensus.cc:2379] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 6: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:46.234520 17368 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.244287 17368 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:46.218719 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:46.218838 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:46.222139 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 45988.9us
I20260807 08:01:46.184825 17499 consensus_queue.cc:260] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:46.258415 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.258857 13805 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:46.260336 17499 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:46.270740 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:46.271718 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:46.278548 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:46.279841 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:46.281366 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:46.281710 17790 log.cc:924] Time spent T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260807 08:01:46.281810 14583 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:46.282419 13811 raft_consensus.cc:1226] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:46.287829 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.293220 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:46.353953 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.381134 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.385294 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:46.385462 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:46.375378 17654 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:46.422601 17654 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:46.380677 17486 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:46.578728 17486 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:46.510334 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:46.579298 13805 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:46.580200 13702 raft_consensus.cc:1226] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:46.580765 13817 raft_consensus.cc:1226] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:46.581070 17641 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:46.581923 17641 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.584616 17641 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:46.554365 13804 raft_consensus.cc:1226] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:46.417366 17242 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:46.561576 13584 raft_consensus.cc:2396] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
W20260807 08:01:46.568248 17537 consensus_meta.cc:287] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.500s user 0.004s sys 0.008s
W20260807 08:01:46.568668 17571 consensus_meta.cc:287] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.654s user 0.016s sys 0.000s
I20260807 08:01:46.644902 17571 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.645767 14622 raft_consensus.cc:2396] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260807 08:01:46.646571 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:46.646976 17571 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:46.569024 17576 consensus_meta.cc:287] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.652s user 0.007s sys 0.010s
I20260807 08:01:46.588331 13590 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:46.647570 13592 raft_consensus.cc:2354] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8.
I20260807 08:01:46.647739 17576 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:46.648490 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.649631 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.649847 17576 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:46.588596 17508 consensus_meta.cc:287] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.536s user 0.008s sys 0.007s
I20260807 08:01:46.650745 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:46.650626 17508 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.651558 13592 raft_consensus.cc:2396] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260807 08:01:46.652607 17508 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:46.653834 13701 raft_consensus.cc:1226] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:46.653654 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.653182 17508 consensus_queue.cc:1046] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260807 08:01:46.656414 14618 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 08:01:46.657783 17508 raft_consensus.cc:3032] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:46.589311 17658 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:46.590358 13811 raft_consensus.cc:1226] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
W20260807 08:01:46.598080 13661 connection.cc:232] RPC call timeout handler was delayed by -1.1638722401112318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
I20260807 08:01:46.599112 14629 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:46.600728 17540 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:46.601121 17429 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:46.601861 17603 consensus_meta.cc:287] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.695s user 0.006s sys 0.013s
I20260807 08:01:46.601899 17218 consensus_queue.cc:1046] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260807 08:01:46.601225 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 302388us
I20260807 08:01:46.602557 13769 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:46.553303 13579 raft_consensus.cc:3032] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:46.686622 13579 raft_consensus.cc:729] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:46.597139 14623 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:46.603588 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.604204 13580 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 08:01:46.604385 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.605137 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.605764 17449 consensus_queue.cc:1046] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:46.605355 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:46.611003 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:46.615989 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:46.574250 17524 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:46.617841 13808 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 08:01:46.618183 17679 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:46.622575 13430 catalog_manager.cc:5495] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 3, leader changed from 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 3 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:46.623605 17654 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:46.628015 17722 consensus_meta.cc:287] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.537s user 0.007s sys 0.007s
I20260807 08:01:46.635526 13540 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:46.658190 17508 raft_consensus.cc:729] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:46.692971 13587 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:46.712074 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:46.699275 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 96817.8us
I20260807 08:01:46.700233 13588 raft_consensus.cc:1226] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:46.706092 13701 raft_consensus.cc:1226] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:46.703235 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:46.706305 17658 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:46.682432 17540 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.706768 17522 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:46.716684 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:46.717010 17522 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:46.719012 17522 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:46.719430 17696 consensus_queue.cc:1046] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:46.703109 17571 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:46.678591 17593 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260807 08:01:46.744480 13811 raft_consensus.cc:1226] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:46.682693 17429 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:46.756606 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 151363us
I20260807 08:01:46.753242 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:46.734040 17722 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:46.738667 17571 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:46.755937 17242 consensus_queue.cc:1046] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260807 08:01:46.769644 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:46.773308 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:46.738246 17218 consensus_queue.cc:1225] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer 7a889001d6494fe5b7cbbe7da1dae302 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260807 08:01:46.753894 13579 consensus_queue.cc:260] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.780646 13769 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:46.782399 13708 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:46.784873 13812 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260807 08:01:46.788738 13579 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:46.779976 14619 raft_consensus.cc:2396] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260807 08:01:46.785358 13812 pending_rounds.cc:76] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 6
I20260807 08:01:46.797802 13812 pending_rounds.cc:97] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260807 08:01:46.798164 13812 raft_consensus.cc:2866] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:46.801580 17485 raft_consensus.cc:2726] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:46.801889 17751 raft_consensus.cc:2741] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260807 08:01:46.802416 13702 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:46.803036 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:46.803907 13703 raft_consensus.cc:2396] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:01:46.803807 17429 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:46.804646 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:46.805178 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:46.805979 13701 raft_consensus.cc:2362] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6. Re-sending same reply.
I20260807 08:01:46.749775 17681 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260807 08:01:46.790130 17643 raft_consensus.cc:3032] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:46.731040 17679 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.781457 17571 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:46.833976 13769 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:46.834539 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 199100us
I20260807 08:01:46.838351 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:46.852463 13540 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:46.821197 17429 consensus_queue.cc:1046] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260807 08:01:46.865387 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:46.858408 13769 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:46.862694 13703 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:46.823268 17643 raft_consensus.cc:729] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:46.872527 13708 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:46.864637 17751 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:46.885066 13588 raft_consensus.cc:1226] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:46.885438 17751 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:46.885942 17751 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:46.867810 17218 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:46.891474 17218 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:46.893620 17735 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.021s sys 0.050s
W20260807 08:01:46.894038 17735 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.910s user 0.021s sys 0.050s
I20260807 08:01:46.893842 17524 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:46.894744 17723 consensus_queue.cc:1046] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:46.895627 17723 raft_consensus.cc:3032] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:46.895952 17723 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:46.901554 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:46.903411 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:46.903991 13811 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 5
I20260807 08:01:46.867674 17696 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 08:01:46.911712 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 59355.1us
I20260807 08:01:46.914546 13807 raft_consensus.cc:1226] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:46.914597 17242 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:46.916034 17242 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.918870 17242 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:46.896783 17723 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.922509 17723 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:46.928676 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:46.933598 13587 raft_consensus.cc:1226] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:46.938053 13661 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:46.941217 13579 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260807 08:01:46.944033 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:46.911298 17757 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:46.946346 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:46.946970 13811 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:46.941642 13579 pending_rounds.cc:76] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: Aborting all ops after (but not including) 5
I20260807 08:01:46.950529 13579 pending_rounds.cc:97] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260807 08:01:46.927635 17654 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:46.950845 13537 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:46.951005 13579 raft_consensus.cc:2866] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:46.951164 13540 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:46.955778 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:46.964715 13587 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:46.967036 14583 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:46.970573 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:46.971325 13661 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:46.946363 17429 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:46.977120 17765 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260807 08:01:46.960191 17696 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:46.969133 17593 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:46.997258 13661 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:46.996098 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 52391.5us
I20260807 08:01:47.001145 13585 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:47.001622 17658 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.007308 17447 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:47.007779 17447 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:47.008137 17447 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.009629 17714 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:47.010077 17714 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:47.010429 17714 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.011570 17751 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.014535 17721 raft_consensus.cc:3032] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:47.014998 17721 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:47.015954 17721 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:46.949085 17242 consensus_queue.cc:1046] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260807 08:01:47.017894 17522 raft_consensus.cc:3032] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:47.018316 17522 raft_consensus.cc:729] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:47.018457 17242 raft_consensus.cc:3032] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:47.019022 17242 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:47.019294 17522 consensus_queue.cc:260] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.020702 17522 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:47.021585 17751 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:47.023910 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1515) took 2605 ms. Trace:
I20260807 08:01:47.024317 13807 rpcz_store.cc:276] 0807 08:01:44.418035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:44.940432 (+522397us) service_pool.cc:227] Handling call
0807 08:01:44.940794 (+ 362us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:46.915047 (+1974253us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:47.023682 (+108635us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b540027bfd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:47.023853 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4499149568}
I20260807 08:01:46.976245 17540 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:47.033795 14583 leader_election.cc:334] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.517s (SENT)
I20260807 08:01:47.034296 14583 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:46.976464 17485 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.035220 13430 catalog_manager.cc:5495] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:46.966856 17655 consensus_queue.cc:1046] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260807 08:01:47.037122 17655 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.038053 17696 raft_consensus.cc:686] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:47.039515 17508 consensus_queue.cc:260] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.041055 17499 raft_consensus.cc:3032] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:47.067962 13809 raft_consensus.cc:1226] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:47.048204 13432 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56598 (request call id 301) took 10173 ms. Trace:
I20260807 08:01:47.037325 13430 catalog_manager.cc:5495] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:01:47.051342 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.006040 17593 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:46.936561 17643 consensus_queue.cc:260] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:47.052618 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 55446.8us
I20260807 08:01:47.052996 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:46.954972 17581 consensus_queue.cc:1035] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
W20260807 08:01:47.055156 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 48543.6us
I20260807 08:01:47.066362 17740 raft_consensus.cc:3032] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:46.975450 17757 raft_consensus.cc:686] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:47.053149 17585 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:46.977160 17218 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.048846 17571 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.020215 17242 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.017414 17721 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:46.991398 17429 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.035689 17485 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.082065 13540 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.061019 17499 raft_consensus.cc:729] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:47.101008 13769 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.076444 17593 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:47.136775 13816 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.003s sys 0.016s
W20260807 08:01:47.138695 13816 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.513s user 0.003s sys 0.016s
I20260807 08:01:47.105877 13430 catalog_manager.cc:5495] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 4, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 4 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:47.090636 17679 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.089823 17740 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:47.099113 17751 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260807 08:01:47.155844 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 57122.9us
I20260807 08:01:47.081475 17765 consensus_queue.cc:1046] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
W20260807 08:01:47.158870 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 57959.7us
I20260807 08:01:47.176911 13430 catalog_manager.cc:5495] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:47.157287 17508 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:47.185958 13430 catalog_manager.cc:5495] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 4, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 77da774f2bb54cf28302d53d0db224ad (127.11.140.65). New cstate: current_term: 4 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:47.157073 17218 raft_consensus.cc:2726] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.194814 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:47.195568 13661 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.157737 17537 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:47.191705 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.197639 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.200884 14629 raft_consensus.cc:2396] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260807 08:01:47.198436 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:47.192610 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 110613us
I20260807 08:01:47.201467 13708 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:47.171837 17489 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:47.207377 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:47.172120 17751 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.177634 17603 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.111181 17643 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.208707 13430 catalog_manager.cc:5495] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:47.193420 17722 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.211886 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.137630 17696 consensus_queue.cc:237] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.068513 13432 rpcz_store.cc:276] 0807 08:01:36.874964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:36.875136 (+ 172us) service_pool.cc:227] Handling call
0807 08:01:47.047910 (+10172774us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:39.808294 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:39.913299 (+105005us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:39.915430 (+ 2131us) write_op.cc:435] Acquired schema lock
0807 08:01:39.917169 (+ 1739us) tablet.cc:660] Decoding operations
0807 08:01:40.000667 (+ 83498us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:40.000746 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:40.000794 (+ 48us) tablet.cc:683] Acquiring locks for 12 operations
0807 08:01:40.149268 (+148474us) tablet.cc:699] Row locks acquired
0807 08:01:40.150667 (+ 1399us) write_op.cc:260] PREPARE: finished
0807 08:01:40.306032 (+155365us) write_op.cc:270] Start()
0807 08:01:40.350800 (+ 44768us) write_op.cc:276] Timestamp: P: 1786089700302844 usec, L: 0
0807 08:01:40.364108 (+ 13308us) op_driver.cc:331] REPLICATION: starting
0807 08:01:40.549150 (+185042us) log.cc:842] Serialized 4843 byte log entry
0807 08:01:43.098314 (+2549164us) op_driver.cc:444] REPLICATION: finished
0807 08:01:43.293933 (+195619us) write_op.cc:301] APPLY: starting
0807 08:01:43.317731 (+ 23798us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:43.426731 (+109000us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:43.443776 (+ 17045us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:43.518806 (+ 75030us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:43.519063 (+ 257us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0807 08:01:43.519106 (+ 43us) write_op.cc:312] APPLY: finished
0807 08:01:43.519428 (+ 322us) log.cc:842] Serialized 89 byte log entry
0807 08:01:43.519593 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:43.519735 (+ 142us) write_op.cc:454] Released schema lock
0807 08:01:43.519785 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:43.520132 (+ 347us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:43.520157 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16405,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":181194,"apply.run_cpu_time_us":4964,"apply.run_wall_time_us":233874,"num_ops":12,"prepare.queue_time_us":128507,"prepare.run_cpu_time_us":91851,"prepare.run_wall_time_us":3226381,"raft.queue_time_us":292593,"raft.run_cpu_time_us":14039,"raft.run_wall_time_us":117743,"replication_time_us":2722246,"thread_start_us":2557627,"threads_started":4,"wal-append.queue_time_us":2109715,"wal-append.run_cpu_time_us":33779,"wal-append.run_wall_time_us":2149740}]]}
I20260807 08:01:47.208632 17537 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.227622 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.240387 14629 raft_consensus.cc:2396] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:01:47.245326 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:47.247035 13703 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:47.251408 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.252075 13540 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.248838 13430 catalog_manager.cc:5495] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 1 to 3, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 3 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:47.216388 17679 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:47.236322 17617 consensus_queue.cc:1046] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260807 08:01:47.252501 17722 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.215575 17751 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.224131 17447 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.258092 17485 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.234477 17585 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260807 08:01:47.214534 17603 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.206817 17499 consensus_queue.cc:260] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.287185 17681 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.291769 17540 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.253752 17654 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260807 08:01:47.248184 17714 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.292730 17765 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:47.286563 14623 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:47.321691 17429 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.321414 17540 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:47.325683 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 73714.4us
W20260807 08:01:47.347898 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 92651.8us
I20260807 08:01:47.235611 17740 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.313403 17571 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.236522 17757 consensus_queue.cc:237] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.330855 17734 raft_consensus.cc:3032] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:47.354784 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.359118 13702 raft_consensus.cc:1261] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 08:01:47.359620 13430 catalog_manager.cc:5495] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
W20260807 08:01:47.366370 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 268225us
I20260807 08:01:47.344241 17603 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:47.305828 17722 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.378815 13817 raft_consensus.cc:2396] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 08:01:47.381814 13540 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.359699 17734 raft_consensus.cc:729] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:47.336242 17681 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.423319 13661 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:47.426712 13817 raft_consensus.cc:1226] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:47.427547 13430 catalog_manager.cc:5495] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 5 to 7, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 7 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:47.430202 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:47.431958 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 50196.3us
I20260807 08:01:47.438783 17585 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:47.414228 17603 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:47.440196 13702 raft_consensus.cc:2362] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8. Re-sending same reply.
I20260807 08:01:47.431541 17447 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:47.443522 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 225959us
I20260807 08:01:47.451304 13661 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.437306 17524 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:47.455796 13705 raft_consensus.cc:1226] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260807 08:01:47.454830 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:47.461225 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:47.463855 13708 raft_consensus.cc:2379] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:47.441691 17540 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.470515 13701 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:47.476449 13769 connection.cc:232] RPC call timeout handler was delayed by -1.2503253309987485s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21480000000000002s for a response
I20260807 08:01:47.479377 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:47.479684 13661 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:47.480415 17820 log.cc:924] Time spent T 00c1526fe2ae48f3a1bcce761ff91f72 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260807 08:01:47.439616 17485 consensus_queue.cc:1046] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260807 08:01:47.481009 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:47.481353 17485 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.481830 17485 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.483649 17765 consensus_queue.cc:1046] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:47.484213 17485 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.464627 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.485878 17722 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:47.487286 17537 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:47.488206 17714 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:47.491514 17593 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.493031 17681 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:47.493443 17593 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:47.462013 17740 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.453912 17734 consensus_queue.cc:260] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.500181 17734 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.469056 17486 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.502022 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.451612 17723 consensus_queue.cc:1046] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260807 08:01:47.503707 17723 raft_consensus.cc:3032] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:47.504067 17723 raft_consensus.cc:729] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:47.506533 17524 consensus_queue.cc:1046] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:47.507637 17524 raft_consensus.cc:3032] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:47.507978 17524 raft_consensus.cc:729] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:47.507347 13430 catalog_manager.cc:5495] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "77da774f2bb54cf28302d53d0db224ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260807 08:01:47.503000 14622 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:47.510667 17585 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.515249 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:47.516656 13708 raft_consensus.cc:1203] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Deduplicated request from leader. Original: 4.2->[5.3-5.3] Dedup: 5.3->[]
I20260807 08:01:47.517300 13708 raft_consensus.cc:1226] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260807 08:01:47.504865 17723 consensus_queue.cc:260] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.450714 17603 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.520198 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.520993 13811 raft_consensus.cc:2379] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:47.521072 13705 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:47.508766 17524 consensus_queue.cc:260] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.463521 17751 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:47.525760 17522 consensus_meta.cc:287] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.505s user 0.006s sys 0.009s
W20260807 08:01:47.532959 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 53363.2us
I20260807 08:01:47.533912 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:47.531315 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.538846 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.544871 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:47.545682 13804 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.538277 13701 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:47.548192 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:47.555943 17485 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.556728 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.557438 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:47.548895 13817 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:01:47.558682 13657 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:47.559172 17681 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:47.559648 17641 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:01:47.560066 17641 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:47.563690 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:47.564756 13710 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260807 08:01:47.559978 13661 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.570572 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.571485 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.561420 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:47.573688 13700 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:47.575601 13709 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:47.565207 13710 pending_rounds.cc:76] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Aborting all ops after (but not including) 6
I20260807 08:01:47.576217 13710 pending_rounds.cc:97] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260807 08:01:47.552130 17723 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.581261 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.552527 17524 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:47.586714 13708 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:47.551702 17714 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:47.576524 13710 raft_consensus.cc:2866] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:47.583999 17681 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:47.565774 17641 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:47.561004 17585 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.592976 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.596848 13700 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:47.597594 13579 raft_consensus.cc:1261] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 08:01:47.597682 17522 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.617486 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.617988 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.621841 17603 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:47.621976 13589 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:47.623294 13589 raft_consensus.cc:1261] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 08:01:47.618468 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.624742 13705 raft_consensus.cc:2379] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 4: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:47.597631 17681 raft_consensus.cc:686] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:47.625285 17815 log.cc:924] Time spent T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.406s user 0.003s sys 0.000s
I20260807 08:01:47.625946 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:47.626169 17681 consensus_queue.cc:237] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.627095 17679 raft_consensus.cc:3032] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:47.627468 17679 raft_consensus.cc:729] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:47.627733 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.628326 13705 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:47.628255 17679 consensus_queue.cc:260] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.619091 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:47.619856 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:47.629863 13702 raft_consensus.cc:2362] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4. Re-sending same reply.
I20260807 08:01:47.627131 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1545) took 1356 ms. Trace:
I20260807 08:01:47.598519 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:47.601910 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.602207 17508 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.604851 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.638525 14629 raft_consensus.cc:2379] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:47.608002 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:47.608327 17817 log.cc:924] Time spent T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.379s user 0.000s sys 0.007s
I20260807 08:01:47.609135 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:47.648064 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1901) took 1078 ms. Trace:
I20260807 08:01:47.610746 17218 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.648432 13805 rpcz_store.cc:276] 0807 08:01:46.569194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:46.580217 (+ 11023us) service_pool.cc:227] Handling call
0807 08:01:46.580604 (+ 387us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:46.611984 (+ 31380us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:46.612010 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:46.612434 (+ 424us) log.cc:842] Serialized 26 byte log entry
0807 08:01:46.929533 (+317099us) raft_consensus.cc:1631] Marking committed up to 2
0807 08:01:46.929593 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:46.929692 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:47.607609 (+677917us) spinlock_profiling.cc:243] Waited 4.86 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:47.647745 (+ 40136us) raft_consensus.cc:1671] finished
0807 08:01:47.647769 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:47.647989 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30948,"raft.queue_time_us":32980,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"spinlock_wait_cycles":11172096,"thread_start_us":316671,"threads_started":1,"wal-append.queue_time_us":583300}
I20260807 08:01:47.613291 17242 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.614703 17486 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.616062 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.616801 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:47.624907 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.561208 17485 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.620527 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.634593 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.652926 13580 raft_consensus.cc:2379] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:47.653462 13811 raft_consensus.cc:2379] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:47.653710 13587 raft_consensus.cc:2396] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:47.595017 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:47.631498 13703 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
W20260807 08:01:47.637367 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 43931.2us
I20260807 08:01:47.640648 17679 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:47.640874 17499 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:47.641305 14627 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:47.620684 17522 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.642102 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.642637 17508 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.644129 17527 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:47.643379 14583 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.645121 14620 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:47.645597 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.645752 13584 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 08:01:47.619205 17658 raft_consensus.cc:3032] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:01:47.629490 17714 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.618911 14623 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:47.646587 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.669950 14583 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.631565 13814 rpcz_store.cc:276] 0807 08:01:46.270431 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:46.570224 (+299793us) service_pool.cc:227] Handling call
0807 08:01:46.570584 (+ 360us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:46.604884 (+ 34300us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:46.604917 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:46.605338 (+ 421us) log.cc:842] Serialized 26 byte log entry
0807 08:01:46.954174 (+348836us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:47.241682 (+287508us) log.cc:842] Serialized 14 byte log entry
0807 08:01:47.243549 (+ 1867us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:47.243688 (+ 139us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:47.624624 (+380936us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:47.626842 (+ 2218us) raft_consensus.cc:1671] finished
0807 08:01:47.626942 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:47.627071 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33869,"prepare.queue_time_us":105307,"prepare.run_cpu_time_us":22,"prepare.run_wall_time_us":22,"raft.queue_time_us":36135,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":235,"spinlock_wait_cycles":28882304,"thread_start_us":590482,"threads_started":2,"wal-append.queue_time_us":597560}
I20260807 08:01:47.652242 13592 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.656165 13710 raft_consensus.cc:1226] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:47.670939 14629 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:47.662129 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.665725 13579 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:47.670140 13706 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:47.673645 13540 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.674791 14627 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:47.678555 13804 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:47.681499 13657 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:47.681419 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.649879 17218 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.664939 17658 raft_consensus.cc:729] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:47.688120 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.690495 13540 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.692214 14629 raft_consensus.cc:2396] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:01:47.655831 17486 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.674753 17833 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.699215 17486 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:47.699841 17765 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:47.701679 13591 raft_consensus.cc:2396] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260807 08:01:47.679857 17643 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:47.703140 14583 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.704046 14583 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.676826 17527 raft_consensus.cc:686] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:47.705856 17722 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:47.705184 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:47.707494 17522 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.707836 17508 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:47.708209 17722 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:47.708424 17833 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.708706 17722 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:47.693300 17617 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:47.708545 17658 consensus_queue.cc:260] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.709244 17722 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:47.709928 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.722635 14583 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.711408 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.711656 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.718434 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:47.710503 13592 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:47.724092 13814 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:01:47.726800 13537 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:47.686883 17721 consensus_meta.cc:287] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.559s user 0.012s sys 0.000s
I20260807 08:01:47.729355 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.723054 14625 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:47.733067 17833 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:47.733750 17833 raft_consensus.cc:686] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:47.734912 17833 consensus_queue.cc:237] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.736795 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.737852 13587 raft_consensus.cc:2396] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260807 08:01:47.738173 14629 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:47.738742 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:47.740650 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 50219.2us
I20260807 08:01:47.741636 13540 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.738692 17508 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.742964 17522 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:01:47.757300 17522 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:47.757725 17522 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:47.745882 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.749405 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:47.749768 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:47.761439 14620 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:47.761770 13589 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.762051 17643 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:47.765636 13587 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:47.769665 14623 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 08:01:47.771101 17485 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.751145 17218 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.752562 17593 consensus_queue.cc:1035] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:47.753134 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.721835 17617 raft_consensus.cc:686] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:47.753988 17697 consensus_queue.cc:1035] T 2c1339f29eb34e05a3d97cef349b97c7 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:47.775527 14626 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:47.756661 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.744498 14583 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.776467 13580 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:01:47.776002 17617 consensus_queue.cc:237] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.777264 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.778095 13580 raft_consensus.cc:2396] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260807 08:01:47.779124 13580 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260807 08:01:47.753924 13661 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.780892 13584 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:47.726861 17368 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.781857 17368 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.783001 17697 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:47.782822 17527 consensus_queue.cc:237] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:47.782848 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.784250 13805 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:47.783598 17593 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260807 08:01:47.785351 14629 raft_consensus.cc:1261] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260807 08:01:47.785761 14629 pending_rounds.cc:76] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Aborting all ops after (but not including) 6
I20260807 08:01:47.785981 14629 pending_rounds.cc:97] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260807 08:01:47.786185 14629 raft_consensus.cc:2866] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:47.788864 14623 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:47.789182 14619 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:47.789378 17714 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:47.790072 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:47.801719 17828 log.cc:924] Time spent T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.217s user 0.002s sys 0.000s
I20260807 08:01:47.802170 17218 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:47.790508 14620 raft_consensus.cc:1261] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260807 08:01:47.801573 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 60073.2us
I20260807 08:01:47.803918 13540 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.788280 14630 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:47.806130 13661 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.807416 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.814812 13589 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:47.817442 13540 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.815811 13587 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:47.818818 13540 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:47.819327 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.820004 13587 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:47.827600 13585 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 08:01:47.828328 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 275018us
I20260807 08:01:47.830214 13540 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.831724 13661 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:47.834420 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 90019.5us
W20260807 08:01:47.835455 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 91689.5us
I20260807 08:01:47.838120 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:47.780709 17641 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.838590 13661 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.844983 14583 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:47.846414 17447 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:47.847078 17447 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:47.847502 17447 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:47.814678 17585 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.848372 13540 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:47.814499 17658 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:47.826694 17581 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:47.853626 14583 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:47.832922 17655 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:47.840158 17485 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 08:01:47.850229 17821 log.cc:924] Time spent T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.533s user 0.006s sys 0.003s
I20260807 08:01:47.844671 17696 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:47.844384 17524 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260807 08:01:47.922950 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 66189.5us
I20260807 08:01:47.844116 14618 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:47.858345 17499 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 08:01:47.955010 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 106724us
I20260807 08:01:47.866907 17740 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:47.922323 17817 log.cc:924] Time spent T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260807 08:01:47.866407 17585 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:47.956755 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:47.922719 17581 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:47.945207 17696 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:47.864475 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.964445 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:47.947611 17655 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:47.945231 17723 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:47.979444 13588 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:47.974675 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.973754 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:47.866663 17765 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.101s
I20260807 08:01:47.972961 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:47.985971 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:48.010259 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:48.011726 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:47.987843 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:47.988327 13812 raft_consensus.cc:1261] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 08:01:47.989105 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:48.013566 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:48.014241 13708 raft_consensus.cc:2379] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 9: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:48.014334 13585 raft_consensus.cc:2354] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:47.989847 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.990289 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.990639 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.991484 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:47.991982 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:47.992403 13701 raft_consensus.cc:1226] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:47.963409 17585 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:47.995471 13540 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:47.957912 17499 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:47.948261 17522 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:47.980227 17841 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:47.968443 17696 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:48.031507 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:48.018883 13706 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:47.974207 17641 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:48.031519 13540 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:48.032913 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 45160.1us
I20260807 08:01:48.033046 14629 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:48.034190 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1913) took 1162 ms. Trace:
I20260807 08:01:48.020413 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:48.035400 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 196895us
W20260807 08:01:48.038049 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 187273us
I20260807 08:01:48.087829 13769 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:48.090113 13592 raft_consensus.cc:2379] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 9: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:48.090636 13588 raft_consensus.cc:2379] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 6: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:47.984895 17723 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:47.969022 17872 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260807 08:01:48.039523 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 159287us
I20260807 08:01:48.040799 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:48.093176 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:48.029369 17429 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:47.966156 17740 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.020501 17655 raft_consensus.cc:686] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:48.118134 13661 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:48.029721 17508 consensus_queue.cc:1035] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260807 08:01:48.119014 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:48.036185 17585 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:48.129038 13805 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:48.130271 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:48.087786 13808 rpcz_store.cc:276] 0807 08:01:46.872226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:46.872394 (+ 168us) service_pool.cc:227] Handling call
0807 08:01:46.872875 (+ 481us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:46.873741 (+ 866us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:46.873804 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:46.874315 (+ 511us) log.cc:842] Serialized 26 byte log entry
0807 08:01:47.194912 (+320597us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:47.194966 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:47.195063 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:47.623334 (+428271us) spinlock_profiling.cc:243] Waited 5.57 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:47.827579 (+204245us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b2400257a98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19e9bcf4 00007f6b19e99dfb 00007f6b19f28559 00007f6b19f281e5 00007f6b19f2782d 00007f6b1cbd646e 00007f6b1cbedc0c 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551
0807 08:01:48.033930 (+206351us) raft_consensus.cc:1671] finished
0807 08:01:48.033962 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:48.034130 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19311,"raft.queue_time_us":1468,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":6552,"spinlock_wait_cycles":76321920,"thread_start_us":304760,"threads_started":1,"wal-append.queue_time_us":366203}
W20260807 08:01:48.128621 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 97194.4us
W20260807 08:01:48.171219 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 53162.5us
I20260807 08:01:48.171707 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:48.172412 13661 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:48.172395 13585 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:48.173753 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:48.174540 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:48.035881 17499 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:48.176584 13590 raft_consensus.cc:2396] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260807 08:01:48.177606 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:48.129390 17723 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:48.169754 17218 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260807 08:01:48.179030 17855 log.cc:924] Time spent T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.223s user 0.000s sys 0.001s
I20260807 08:01:48.179502 17641 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:48.180084 17641 raft_consensus.cc:686] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:48.180380 17862 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:48.180857 17862 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:48.182503 17681 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:48.181257 17862 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:48.165757 17751 raft_consensus.cc:2726] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:48.170727 17833 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:48.186525 17833 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.122772 14620 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:48.177480 17885 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:48.192884 13540 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:48.228695 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 140939us
I20260807 08:01:48.233218 13769 leader_election.cc:398] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8.
I20260807 08:01:48.230084 13805 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 9.
I20260807 08:01:48.134744 17872 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:48.233623 13769 leader_election.cc:401] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260807 08:01:48.233911 17872 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:48.235291 13769 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:48.236138 17734 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8.
I20260807 08:01:48.236835 17734 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:48.238150 13769 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:48.239074 17429 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:01:48.242758 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 49913us
W20260807 08:01:48.188691 17860 log.cc:924] Time spent T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.175s user 0.000s sys 0.000s
I20260807 08:01:48.248062 13585 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:48.175949 17765 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:48.078191 17841 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.251217 13769 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:48.129971 17581 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.098s
I20260807 08:01:48.248315 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:48.259090 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:48.270519 13585 raft_consensus.cc:2354] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:48.261072 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:48.193454 17527 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:48.120724 17869 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.162s
I20260807 08:01:48.271449 13585 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:48.242183 17508 consensus_queue.cc:1046] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260807 08:01:48.242496 17537 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:48.258137 17218 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:48.241621 17485 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:48.180974 17641 consensus_queue.cc:237] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:48.314275 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 140228us
I20260807 08:01:48.318708 13706 raft_consensus.cc:1261] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260807 08:01:48.321095 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 50754us
I20260807 08:01:48.318320 14626 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:48.341831 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:48.364929 13769 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:48.267755 17833 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:48.343269 13814 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:48.344904 13585 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260807 08:01:48.342489 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 88013.8us
I20260807 08:01:48.369433 13585 raft_consensus.cc:1261] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260807 08:01:48.362277 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 104236us
I20260807 08:01:48.381274 13657 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:48.341943 17751 raft_consensus.cc:2726] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:48.272035 17765 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:48.391613 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:48.392427 13814 raft_consensus.cc:2379] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 9: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:48.393200 13805 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:48.360176 17218 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:48.399216 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:48.403843 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 155649us
I20260807 08:01:48.384841 17522 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:48.409670 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:48.425170 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:48.412576 13591 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:01:48.379259 17734 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 08:01:48.458952 17447 consensus_meta.cc:287] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.605s user 0.010s sys 0.000s
I20260807 08:01:48.407181 17218 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:48.444991 13808 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 8: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:48.444325 13805 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:48.387486 17885 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 08:01:48.446461 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 47339.9us
I20260807 08:01:48.363783 17537 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.416188 17679 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election won for term 9
W20260807 08:01:48.492686 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 116912us
I20260807 08:01:48.359444 17527 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.388185 17508 raft_consensus.cc:3032] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:48.300163 17655 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.459276 17751 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:48.460142 17740 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:48.551308 13769 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:48.510946 17734 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:48.552912 17722 consensus_meta.cc:287] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.802s user 0.011s sys 0.000s
I20260807 08:01:48.570426 17527 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:48.560034 17508 raft_consensus.cc:729] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:48.573675 17881 log.cc:924] Time spent T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
I20260807 08:01:48.574600 17508 consensus_queue.cc:260] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:48.576309 17508 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:48.579005 17537 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:48.544773 17885 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:48.561405 17895 log.cc:924] Time spent T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260807 08:01:48.583683 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 202443us
W20260807 08:01:48.596776 14622 consensus_meta.cc:287] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.512s user 0.009s sys 0.000s
I20260807 08:01:48.598776 13661 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:48.599124 14622 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:48.602448 13661 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:48.568951 17722 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.605181 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:48.607055 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:48.607967 13814 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:48.610237 14580 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:48.611763 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:48.583644 17734 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:48.521898 17447 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.598750 17885 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:48.582993 17902 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:48.633770 17883 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:48.634086 17902 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.634353 17883 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.636092 17901 consensus_queue.cc:1035] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.139s
I20260807 08:01:48.636535 17902 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:48.622881 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:48.639354 13584 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:01:48.623108 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:48.623564 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 199119us
I20260807 08:01:48.583335 17679 raft_consensus.cc:686] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:48.629245 14627 raft_consensus.cc:1226] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:48.608665 17907 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260807 08:01:48.632736 17916 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:48.633245 17888 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:48.641901 13585 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260807 08:01:48.651225 13585 pending_rounds.cc:76] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Aborting all ops after (but not including) 6
I20260807 08:01:48.651566 13585 pending_rounds.cc:97] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260807 08:01:48.651871 13585 raft_consensus.cc:2866] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:48.642783 14583 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:48.652050 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:48.654057 13701 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
W20260807 08:01:48.645253 17658 consensus_meta.cc:287] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.790s user 0.007s sys 0.004s
I20260807 08:01:48.663923 13540 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:48.628679 17841 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:48.670265 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:48.655350 17242 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:48.661119 17641 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:01:48.665633 17882 log.cc:924] Time spent T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.345s user 0.001s sys 0.000s
I20260807 08:01:48.717247 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:48.718297 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:48.720584 13588 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:48.654887 17751 raft_consensus.cc:686] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:48.723213 13701 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:48.674921 17485 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260807 08:01:48.733404 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 120333us
I20260807 08:01:48.738040 13661 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:48.737723 17485 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:48.724368 17765 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:48.753854 13580 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.000s sys 0.017s
I20260807 08:01:48.715179 17242 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:48.765586 13580 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.649s user 0.001s sys 0.017s
I20260807 08:01:48.717875 17909 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 08:01:48.765626 17871 log.cc:924] Time spent T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.334s user 0.001s sys 0.000s
I20260807 08:01:48.774432 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:48.776513 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 131734us
W20260807 08:01:48.790421 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 267816us
I20260807 08:01:48.790684 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:48.790550 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1862) took 1045 ms. Trace:
I20260807 08:01:48.765883 17485 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:48.805434 13592 rpcz_store.cc:276] 0807 08:01:47.745285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:47.745607 (+ 322us) service_pool.cc:227] Handling call
0807 08:01:48.790346 (+1044739us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b54001356d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:48.790487 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":742265088}
I20260807 08:01:48.770010 17242 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:48.788558 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:48.714246 17913 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260807 08:01:48.794008 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:48.771157 17883 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:48.774993 17909 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:48.805678 17696 consensus_meta.cc:287] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.725s user 0.010s sys 0.002s
I20260807 08:01:48.786652 17522 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:48.661454 17916 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.812543 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 576) took 1024 ms. Trace:
I20260807 08:01:48.791759 13769 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:48.817927 13584 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:01:48.819247 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 81295.6us
I20260807 08:01:48.769626 17486 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
W20260807 08:01:48.833573 14630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 553) took 1284 ms (client timeout 1639 ms). Trace:
W20260807 08:01:48.804468 17875 log.cc:924] Time spent T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.366s user 0.000s sys 0.001s
I20260807 08:01:48.764755 17679 consensus_queue.cc:237] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:48.850905 17916 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:48.840749 17522 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:48.851794 17522 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:01:48.843837 14630 rpcz_store.cc:269] 0807 08:01:47.549639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:47.549955 (+ 316us) service_pool.cc:227] Handling call
0807 08:01:48.833380 (+1283425us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54006054d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:48.833488 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":401700736}
I20260807 08:01:48.791728 17915 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260807 08:01:48.840449 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 63105us
I20260807 08:01:48.874071 13540 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:48.832259 17722 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:48.876871 17912 log.cc:924] Time spent T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.225s user 0.001s sys 0.000s
I20260807 08:01:48.841183 14625 rpcz_store.cc:276] 0807 08:01:47.788064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:47.790569 (+ 2505us) service_pool.cc:227] Handling call
0807 08:01:47.790886 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:47.826079 (+ 35193us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:47.826122 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:47.859432 (+ 33310us) log.cc:842] Serialized 26 byte log entry
0807 08:01:48.034840 (+175408us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:48.034890 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:48.034963 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:48.812292 (+777329us) raft_consensus.cc:1671] finished
0807 08:01:48.812310 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:48.812466 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34732,"raft.queue_time_us":41957,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":8319,"thread_start_us":77128,"threads_started":1,"wal-append.queue_time_us":305759}
I20260807 08:01:48.886322 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1869) took 1075 ms. Trace:
I20260807 08:01:48.833307 17909 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:48.875612 13584 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:01:48.888011 13580 rpcz_store.cc:276] 0807 08:01:47.810964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:47.811120 (+ 156us) service_pool.cc:227] Handling call
0807 08:01:47.811639 (+ 519us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:47.829400 (+ 17761us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:47.829444 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:47.829969 (+ 525us) log.cc:842] Serialized 26 byte log entry
0807 08:01:48.088613 (+258644us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:48.768043 (+679430us) log.cc:842] Serialized 14 byte log entry
0807 08:01:48.768258 (+ 215us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:48.768342 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:48.886095 (+117753us) raft_consensus.cc:1671] finished
0807 08:01:48.886117 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:48.886238 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37999,"prepare.queue_time_us":281455,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":26331,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":12422,"thread_start_us":805491,"threads_started":2,"wal-append.queue_time_us":487328}
I20260807 08:01:48.840328 17910 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:48.846486 17447 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:48.890471 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 45914.2us
I20260807 08:01:48.878947 17916 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:48.939261 14583 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:48.875834 17883 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election won for term 4
W20260807 08:01:48.887537 17879 log.cc:924] Time spent T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.375s user 0.000s sys 0.001s
I20260807 08:01:48.940541 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1870) took 1124 ms. Trace:
W20260807 08:01:48.884238 17723 consensus_meta.cc:287] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.706s user 0.007s sys 0.002s
I20260807 08:01:48.863153 17915 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:48.864784 17368 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:48.840797 17696 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.951223 13579 rpcz_store.cc:276] 0807 08:01:47.815784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:47.815925 (+ 141us) service_pool.cc:227] Handling call
0807 08:01:47.843189 (+ 27264us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:48.121397 (+278208us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:48.121433 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:48.133814 (+ 12381us) log.cc:842] Serialized 26 byte log entry
0807 08:01:48.315726 (+181912us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:48.458565 (+142839us) log.cc:842] Serialized 14 byte log entry
0807 08:01:48.940228 (+481663us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:48.940313 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:48.940355 (+ 42us) raft_consensus.cc:1671] finished
0807 08:01:48.940365 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:48.940486 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":108439,"prepare.queue_time_us":171890,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":209625,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":20715,"spinlock_wait_cycles":16128,"thread_start_us":888405,"threads_started":4,"wal-append.queue_time_us":130504}
I20260807 08:01:48.954443 17924 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:48.954909 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:48.955698 13804 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:01:48.955860 17921 log.cc:924] Time spent T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20260807 08:01:48.939707 17927 raft_consensus.cc:2726] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 08:01:48.942875 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:48.970544 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:48.877318 17218 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.947217 17447 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:01:48.914073 17734 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:48.948443 17916 raft_consensus.cc:686] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:49.041296 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 249612us
W20260807 08:01:49.040746 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 97908.7us
I20260807 08:01:48.863785 17751 consensus_queue.cc:237] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:48.941731 17540 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:49.035113 17655 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
W20260807 08:01:49.030047 17871 log.cc:924] Time spent T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260807 08:01:48.967993 17883 raft_consensus.cc:686] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:48.938761 17910 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.957784 17723 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:48.970957 17937 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.475s
I20260807 08:01:49.081393 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.170670 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 562) took 1926 ms. Trace:
W20260807 08:01:49.939100 13814 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.008s sys 0.011s
W20260807 08:01:49.633358 14622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 551) took 2118 ms (client timeout 1601 ms). Trace:
I20260807 08:01:50.222522 13769 leader_election.cc:398] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:48.966466 17924 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.176059 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 563) took 1923 ms. Trace:
W20260807 08:01:50.203235 13653 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.003s sys 0.016s
W20260807 08:01:50.231921 13653 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.312s user 0.004s sys 0.017s
I20260807 08:01:48.954052 17915 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:49.832357 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56576 (request call id 1000) took 1207 ms. Trace:
W20260807 08:01:50.224092 14622 rpcz_store.cc:269] 0807 08:01:47.514838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:47.514986 (+ 148us) service_pool.cc:227] Handling call
0807 08:01:49.633173 (+2118187us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b540066c6d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:49.633288 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1308391168}
I20260807 08:01:50.006809 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1542) took 1597 ms. Trace:
I20260807 08:01:50.006237 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 581) took 1382 ms. Trace:
I20260807 08:01:50.237226 13585 rpcz_store.cc:276] 0807 08:01:48.623844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.623992 (+ 148us) service_pool.cc:227] Handling call
0807 08:01:48.624351 (+ 359us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:48.652477 (+ 28126us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:50.005890 (+1353413us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b54002053d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:50.006153 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39506176}
W20260807 08:01:49.971896 13804 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.004s sys 0.016s
W20260807 08:01:50.238992 13804 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s user 0.006s sys 0.022s
I20260807 08:01:50.222905 14627 rpcz_store.cc:276] 0807 08:01:48.244340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.244494 (+ 154us) service_pool.cc:227] Handling call
0807 08:01:48.244765 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:48.645759 (+400994us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:50.170398 (+1524639us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b54003354d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:50.170593 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":883613952}
I20260807 08:01:49.634119 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:49.076705 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 104316us
I20260807 08:01:50.240031 13811 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:49.970521 13537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.000s sys 0.009s
W20260807 08:01:50.238795 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 256450us
W20260807 08:01:50.223652 13814 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.941s user 0.008s sys 0.021s
W20260807 08:01:49.835018 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 559) took 1845 ms (client timeout 1897 ms). Trace:
I20260807 08:01:49.395143 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1872) took 1577 ms. Trace:
W20260807 08:01:49.713904 17971 log.cc:924] Time spent T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Append to log took a long time: real 0.092s user 0.000s sys 0.002s
W20260807 08:01:49.957032 17885 consensus_meta.cc:287] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.312s user 0.012s sys 0.000s
W20260807 08:01:49.077481 17429 consensus_meta.cc:287] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.838s user 0.009s sys 0.003s
I20260807 08:01:50.233031 13589 rpcz_store.cc:276] 0807 08:01:48.624819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.632042 (+ 7223us) service_pool.cc:227] Handling call
0807 08:01:49.831559 (+1199517us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b54001e4e54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0af0 00007f6b1cf73f2a 00007f6b21eb0c6e 00007f6b1b5a7807 00007f6b1b5a7792 00007f6b1b5a76c1 00007f6b1b5a763c 00007f6b1b5a68f2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:49.832280 (+ 721us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":505844608}
W20260807 08:01:50.192400 17947 log.cc:924] Time spent T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.499s user 0.001s sys 0.001s
I20260807 08:01:50.242420 13769 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:49.080353 17218 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.243854 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:50.234913 13710 rpcz_store.cc:276] 0807 08:01:48.409298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.409426 (+ 128us) service_pool.cc:227] Handling call
0807 08:01:50.006577 (+1597151us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54001dc4d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:50.006741 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1403248384}
I20260807 08:01:49.062606 17920 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:49.078109 17933 log.cc:924] Time spent T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.188s user 0.000s sys 0.000s
W20260807 08:01:49.503289 17941 log.cc:924] Time spent T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.422s user 0.000s sys 0.001s
W20260807 08:01:49.867300 17485 consensus_meta.cc:287] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.035s user 0.008s sys 0.003s
W20260807 08:01:49.901494 17522 consensus_meta.cc:287] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.049s user 0.012s sys 0.000s
I20260807 08:01:49.780210 17844 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.269670 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:49.657230 17696 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:49.394199 17927 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.313629 17740 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:49.714752 17641 consensus_meta.cc:287] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.992s user 0.010s sys 0.001s
I20260807 08:01:49.246969 17950 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:49.780936 17585 consensus_meta.cc:287] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.590s user 0.009s sys 0.004s
I20260807 08:01:50.113628 17977 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.354580 17585 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:50.218871 17909 consensus_meta.cc:287] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 1.272s user 0.009s sys 0.003s
I20260807 08:01:50.231705 14629 rpcz_store.cc:276] 0807 08:01:48.252570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.252913 (+ 343us) service_pool.cc:227] Handling call
0807 08:01:48.253305 (+ 392us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:48.355786 (+102481us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:48.355828 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:48.381637 (+ 25809us) log.cc:842] Serialized 26 byte log entry
0807 08:01:48.603894 (+222257us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:48.719570 (+115676us) log.cc:842] Serialized 14 byte log entry
0807 08:01:48.719819 (+ 249us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:48.719905 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:50.173169 (+1453264us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b18003e7f40. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:01:50.175692 (+ 2523us) raft_consensus.cc:1671] finished
0807 08:01:50.175827 (+ 135us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:50.175982 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30915,"prepare.queue_time_us":281806,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":98911,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":18706,"spinlock_wait_cycles":27671424,"thread_start_us":366020,"threads_started":3,"wal-append.queue_time_us":332719}
W20260807 08:01:50.293711 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 57472.5us
I20260807 08:01:50.294179 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:50.351613 17836 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:50.290805 14626 rpcz_store.cc:269] 0807 08:01:47.989909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:47.990031 (+ 122us) service_pool.cc:227] Handling call
0807 08:01:49.834806 (+1844775us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b540043c6d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:49.834943 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":688384256}
I20260807 08:01:49.173851 17447 raft_consensus.cc:686] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 8 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
W20260807 08:01:49.778874 14618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 719) took 2314 ms (client timeout 1577 ms). Trace:
W20260807 08:01:50.267396 13537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.846s user 0.003s sys 0.016s
I20260807 08:01:50.328226 17721 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.028s
I20260807 08:01:50.305344 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:50.365661 13701 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:49.845942 17757 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:49.846931 17242 consensus_meta.cc:287] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 1.014s user 0.009s sys 0.001s
W20260807 08:01:49.241879 17872 consensus_meta.cc:287] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.007s user 0.019s sys 0.001s
W20260807 08:01:49.853580 17949 log.cc:924] Time spent T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.352s user 0.004s sys 0.000s
W20260807 08:01:49.712137 17956 log.cc:924] Time spent T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260807 08:01:50.359220 18009 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.311848 13706 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:50.289192 17888 consensus_meta.cc:287] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.607s user 0.013s sys 0.000s
I20260807 08:01:50.313931 13816 raft_consensus.cc:1261] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 08:01:50.352650 17740 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.314066 13585 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260807 08:01:49.512229 17862 consensus_meta.cc:287] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.327s user 0.013s sys 0.000s
W20260807 08:01:49.936055 17765 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.507s user 0.001s sys 0.014s
W20260807 08:01:49.485432 17499 consensus_meta.cc:287] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.308s user 0.009s sys 0.005s
I20260807 08:01:49.700598 17603 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:49.505470 17508 consensus_meta.cc:287] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.929s user 0.012s sys 0.000s
I20260807 08:01:50.268500 17893 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.356323 17916 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.409543 17893 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.411686 17603 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.413771 17734 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:50.414304 17740 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:50.414511 17734 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.414942 17734 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.416857 17734 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:50.250445 17524 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:49.676645 17938 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.425093 17893 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.424988 17938 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.327855 13807 raft_consensus.cc:3032] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:50.427938 13807 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:50.375029 17897 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.289625 17983 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.241390 17540 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.315577 17833 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.319209 17920 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.318966 17486 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260807 08:01:50.328361 17485 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.230965 17655 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.269266 17449 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.339464 17981 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.340233 18004 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.265347 17924 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.341818 17918 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:50.344961 14627 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:50.311439 17429 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:49.513705 17926 consensus_queue.cc:1035] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 1.263s
I20260807 08:01:50.354693 18010 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:49.453333 17658 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:50.352380 13540 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.68:40123 timed out after 1.601s (SENT) [suppressed 27 similar messages]
I20260807 08:01:50.266873 13587 rpcz_store.cc:276] 0807 08:01:47.817549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:47.820708 (+ 3159us) service_pool.cc:227] Handling call
0807 08:01:47.821082 (+ 374us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:47.839829 (+ 18747us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:47.839862 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:47.842861 (+ 2999us) log.cc:842] Serialized 26 byte log entry
0807 08:01:48.091422 (+248561us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:48.091464 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:48.091531 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:49.394883 (+1303352us) raft_consensus.cc:1671] finished
0807 08:01:49.394908 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:49.395074 (+ 166us) inbound_call.cc:173] Queueing success response
0807 08:01:49.779803 (+384729us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b48000cd758. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b21e127f1 00007f6b21e098e7 00007f6b21e0977a 00007f6b21efd95f 00007f6b21efd627 00007f6b21f11b2c 00007f6b21f11ad8 00007f6b21f11a58 00007f6b21f11a13 00007f6b21f10cbc 00007f6b20e92be6
Metrics: {"mutex_wait_us":8836,"raft.queue_time_us":20354,"raft.run_cpu_time_us":25263,"raft.run_wall_time_us":1936317,"spinlock_wait_cycles":4917632,"thread_start_us":141592,"threads_started":1,"wal-append.queue_time_us":344847}
I20260807 08:01:50.307116 17885 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.346590 17218 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:50.358103 13661 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:50.570988 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 212962us
I20260807 08:01:50.571933 13661 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:50.358039 14622 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:49.654501 17934 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.577960 17934 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.580185 17934 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:50.353214 17696 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:50.363226 14618 rpcz_store.cc:269] 0807 08:01:47.464840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:47.467869 (+ 3029us) service_pool.cc:227] Handling call
0807 08:01:49.778637 (+2310768us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b540020dad4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:49.778785 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":349906176}
I20260807 08:01:50.580806 17696 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.365140 13812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1565) took 2173 ms. Trace:
I20260807 08:01:50.581570 13812 rpcz_store.cc:276] 0807 08:01:48.192129 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.192291 (+ 162us) service_pool.cc:227] Handling call
0807 08:01:48.192608 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:48.241707 (+ 49099us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:48.241749 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:48.242185 (+ 436us) log.cc:842] Serialized 26 byte log entry
0807 08:01:48.739774 (+497589us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:48.739830 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:48.739936 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:50.364916 (+1624980us) raft_consensus.cc:1671] finished
0807 08:01:50.364943 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:50.365076 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":37976,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":12090,"thread_start_us":372686,"threads_started":1,"wal-append.queue_time_us":541055}
I20260807 08:01:50.582815 17696 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.370533 14626 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:55594
I20260807 08:01:50.370826 17723 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.346288 17522 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.585299 17522 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:50.310504 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 43746.5us
I20260807 08:01:49.803084 17958 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.354781 17977 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.357192 17986 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:50.371337 17242 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.373821 18005 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.405614 17862 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.561831 17918 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.641999 17920 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.568406 18010 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.643043 17918 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.407675 17765 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.644780 18010 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.645174 17765 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.408244 17499 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.375535 17757 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.410257 17585 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.417304 17984 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.390460 17447 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.364149 17836 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.417414 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:50.391074 17888 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:50.419581 17740 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:01:50.420959 13701 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:50.423027 17603 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:49.889559 17907 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.423734 17508 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:49.864511 17967 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:49.078413 14620 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:50.327384 13811 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:50.347877 17844 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.357548 17909 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.428741 13807 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.353381 17927 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.391911 18009 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:50.433048 17988 log.cc:924] Time spent T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260807 08:01:50.433583 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1545) took 1769 ms. Trace:
I20260807 08:01:50.436125 13706 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:50.398545 18001 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.439831 17983 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.424855 17524 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.361989 17883 consensus_queue.cc:237] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.462568 17833 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.487462 17485 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.532269 18016 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.532433 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1948) took 1581 ms. Trace:
W20260807 08:01:50.462009 17949 log.cc:924] Time spent T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260807 08:01:50.435397 17734 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:50.359723 17950 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.376863 17872 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.561560 13805 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
W20260807 08:01:50.564428 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 119652us
W20260807 08:01:50.565552 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 323170us
I20260807 08:01:50.533159 17910 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.576668 17916 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.553212 18004 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.645455 17540 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.667484 13807 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:50.649727 18005 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.581166 13808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 605) took 1828 ms. Trace:
I20260807 08:01:50.668236 13808 rpcz_store.cc:276] 0807 08:01:48.753205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.753440 (+ 235us) service_pool.cc:227] Handling call
0807 08:01:48.753818 (+ 378us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:48.887539 (+133721us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:48.887573 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:48.889589 (+ 2016us) log.cc:842] Serialized 26 byte log entry
0807 08:01:49.371349 (+481760us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:49.371409 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:49.371515 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:50.580969 (+1209454us) raft_consensus.cc:1671] finished
0807 08:01:50.580991 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:50.581111 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":115893,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":18223,"thread_start_us":614586,"threads_started":2,"wal-append.queue_time_us":1541440}
I20260807 08:01:50.583387 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:50.357321 17641 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:50.669732 17907 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.670882 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:50.671223 18009 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.588953 13585 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260807 08:01:50.672250 17907 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.672839 18005 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.643071 17862 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.438830 17897 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.674518 17909 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.675727 17927 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:50.676123 14620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 565) took 1818 ms (client timeout 1577 ms). Trace:
I20260807 08:01:50.648617 17990 consensus_queue.cc:1035] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
W20260807 08:01:50.676672 14620 rpcz_store.cc:269] 0807 08:01:48.857402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.885634 (+ 28232us) service_pool.cc:227] Handling call
0807 08:01:50.676042 (+1790408us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:50.676606 13814 rpcz_store.cc:276] 0807 08:01:48.950802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.950946 (+ 144us) service_pool.cc:227] Handling call
0807 08:01:48.956504 (+ 5558us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:48.956711 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 08:01:49.281944 (+325233us) log.cc:842] Serialized 14 byte log entry
0807 08:01:50.532037 (+1250093us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:50.532102 (+ 65us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:50.532159 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:50.532233 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:50.532373 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5202,"prepare.queue_time_us":1311671,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"raft.queue_time_us":124225,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":1569253,"threads_started":3,"wal-append.queue_time_us":1042394}
I20260807 08:01:50.678593 13804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1950) took 1721 ms. Trace:
I20260807 08:01:50.679467 17836 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.680660 17950 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.679570 13708 rpcz_store.cc:276] 0807 08:01:48.664171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.664306 (+ 135us) service_pool.cc:227] Handling call
0807 08:01:48.675474 (+ 11168us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:48.675938 (+ 464us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:48.675975 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:48.709459 (+ 33484us) log.cc:842] Serialized 52 byte log entry
0807 08:01:48.814144 (+104685us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:48.892413 (+ 78269us) log.cc:842] Serialized 14 byte log entry
0807 08:01:48.903910 (+ 11497us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:48.904013 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:50.433317 (+1529304us) raft_consensus.cc:1671] finished
0807 08:01:50.433335 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:50.433506 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83018,"prepare.queue_time_us":64342,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":210313,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":14821,"thread_start_us":93382,"threads_started":2,"wal-append.queue_time_us":200758}
I20260807 08:01:50.665304 14627 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:50.675818 17897 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:50.682312 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 110432us
I20260807 08:01:50.682489 17617 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:50.683001 17757 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:50.683343 13661 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:50.683722 17984 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:50.685128 13805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1569) took 1967 ms (client timeout 2244 ms). Trace:
I20260807 08:01:50.685225 17524 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.566256 17981 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:50.685683 13805 rpcz_store.cc:269] 0807 08:01:48.717920 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.718058 (+ 138us) service_pool.cc:227] Handling call
0807 08:01:50.684980 (+1966922us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54001b6a54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:50.685074 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4227587456}
I20260807 08:01:50.686863 17984 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.687399 17981 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.687245 17869 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:50.681409 13817 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:50.626364 18013 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.695394 14618 raft_consensus.cc:1226] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:50.639547 17973 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.641124 17885 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.719108 14626 raft_consensus.cc:595] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:50.673377 17872 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.652937 13816 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 08:01:50.689725 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1571) took 1056 ms. Trace:
I20260807 08:01:50.637032 14622 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:50.691699 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:50.680701 13769 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:50.576579 17486 consensus_queue.cc:1035] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:50.540191 17449 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.628675 17655 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.695384 13580 raft_consensus.cc:1226] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260807 08:01:50.585639 17893 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:50.678956 13804 rpcz_store.cc:276] 0807 08:01:48.957568 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.957869 (+ 301us) service_pool.cc:227] Handling call
0807 08:01:49.036979 (+ 79110us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:49.076067 (+ 39088us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:49.076096 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:49.265782 (+189686us) log.cc:842] Serialized 26 byte log entry
0807 08:01:50.239370 (+973588us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:50.239395 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:50.239445 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:50.678395 (+438950us) raft_consensus.cc:1671] finished
0807 08:01:50.678417 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:50.678537 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":155543,"thread_start_us":797090,"threads_started":1,"wal-append.queue_time_us":979645}
I20260807 08:01:50.688364 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:50.667110 18016 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.766866 17833 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.769503 13585 raft_consensus.cc:2379] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 9: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:50.687561 17967 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.726358 13579 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:50.705317 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:50.726380 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 373994us
I20260807 08:01:50.757344 17907 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.569224 17429 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:50.662745 17242 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.728886 13811 rpcz_store.cc:276] 0807 08:01:49.633658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:49.633793 (+ 135us) service_pool.cc:227] Handling call
0807 08:01:50.689665 (+1055872us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:50.758361 13588 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
W20260807 08:01:50.772243 13817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1570) took 1761 ms (client timeout 1577 ms). Trace:
I20260807 08:01:50.571554 17658 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.597473 17522 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.774878 14627 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:01:50.725965 14626 raft_consensus.cc:603] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:50.777347 18012 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.732744 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:50.655668 17924 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.734630 13590 raft_consensus.cc:1226] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:50.586552 17218 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.675648 17916 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.653877 17938 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.699800 17508 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.790946 13580 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260807 08:01:50.830639 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 150049us
I20260807 08:01:50.791072 13585 raft_consensus.cc:1261] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 08:01:50.790004 17885 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:50.789669 13817 rpcz_store.cc:269] 0807 08:01:49.011324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:49.081135 (+ 69811us) service_pool.cc:227] Handling call
0807 08:01:50.772058 (+1690923us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54001fdbd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:50.772171 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1055592192}
I20260807 08:01:50.794724 13592 raft_consensus.cc:1261] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 08:01:50.805121 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.832929 13811 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 3.
I20260807 08:01:50.805732 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.806466 13584 raft_consensus.cc:1226] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 08:01:50.756896 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:50.817035 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:50.804375 17862 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.817384 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:50.781090 17603 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.828852 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:01:50.789971 13661 connection.cc:232] RPC call timeout handler was delayed by -1.6379979529883713s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
I20260807 08:01:50.831192 13769 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:50.833034 13807 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260807 08:01:50.771436 17654 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.793728 17918 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.830950 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:50.836549 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:50.888548 13817 raft_consensus.cc:1226] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:50.838563 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:50.892750 13807 pending_rounds.cc:76] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 6
I20260807 08:01:50.893065 13807 pending_rounds.cc:97] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260807 08:01:50.814122 17757 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:50.771766 17836 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.851055 13809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1952) took 1164 ms. Trace:
I20260807 08:01:50.830065 14619 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:50.785602 17907 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.842370 13540 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:50.914076 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.843003 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:50.914670 13817 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:50.847792 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.846956 13588 raft_consensus.cc:2379] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:50.849978 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.915107 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:50.915522 13816 raft_consensus.cc:2396] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260807 08:01:50.792439 17429 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.793387 17893 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:50.916236 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 76427.3us
I20260807 08:01:50.848949 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:50.820667 18014 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.845253 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:50.895586 13809 rpcz_store.cc:276] 0807 08:01:49.686730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:49.686937 (+ 207us) service_pool.cc:227] Handling call
0807 08:01:50.229817 (+542880us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:50.816133 (+586316us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:50.816176 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:50.816227 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:50.816288 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:50.850847 (+ 34559us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b54003c4bd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:50.850894 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:50.851001 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1347805952}
I20260807 08:01:50.897531 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:50.867590 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:50.868055 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:50.855809 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:50.926213 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:50.899677 13702 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:50.927306 13710 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:50.900326 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:50.901216 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:50.899240 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:50.905339 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:50.909957 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.639020 17986 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:50.917570 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 88833.8us
I20260807 08:01:50.918085 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.866693 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:50.893364 13807 raft_consensus.cc:2866] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:50.918428 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:50.844988 17981 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.840219 17524 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:50.948583 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:50.948947 17981 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:50.949782 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 118628us
I20260807 08:01:50.951006 17981 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.951679 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:50.889142 17950 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:50.952610 17950 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.953418 17916 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.953770 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.954375 13809 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:50.954741 17218 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.913796 17603 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.955086 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.948657 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:50.841143 18004 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.866990 17696 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:50.956534 17907 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:50.949178 17524 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.932900 17893 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.957005 17696 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.958395 17524 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.958700 17893 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.956480 13705 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:50.960188 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:50.926781 13584 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:01:50.926231 13703 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:50.667163 18001 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:50.939481 18028 log.cc:924] Time spent T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260807 08:01:50.914747 13706 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:50.689836 17958 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.928747 13540 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:50.994809 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 66138.3us
I20260807 08:01:50.996038 14627 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:50.995990 13540 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:50.729854 17973 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.929857 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:50.930701 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:50.931363 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:50.999811 13588 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:50.841718 17885 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.934274 14580 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:51.003288 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 69091.2us
I20260807 08:01:51.004171 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.004477 14580 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:50.940717 14583 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:51.005153 13584 raft_consensus.cc:2379] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 7: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:50.940939 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:51.005913 13701 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:50.941936 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:50.818320 17522 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.956565 17603 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.915681 13814 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:50.829895 18012 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.920400 13537 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:50.963543 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.014734 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:51.015195 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 94928.5us
I20260807 08:01:51.015834 13703 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:50.866735 17862 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:50.956444 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.975235 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.960146 14626 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:50.953354 13661 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:50.983870 13580 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:50.725687 18013 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.952239 14622 raft_consensus.cc:2362] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6. Re-sending same reply.
I20260807 08:01:50.959261 17696 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:50.986599 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:50.987512 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.988673 13769 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:50.850597 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:50.989950 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.029135 13812 raft_consensus.cc:2379] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:51.029198 13585 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:50.890187 17977 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:51.026360 13706 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:01:50.924880 17486 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:50.954720 17915 consensus_meta.cc:287] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.665s user 0.010s sys 0.000s
I20260807 08:01:50.997530 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:50.935192 17984 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 08:01:50.933259 17985 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.156338 17985 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.156725 17985 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:50.933199 17927 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:50.999887 13592 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:51.008206 13540 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:01:51.161413 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 153311us
I20260807 08:01:51.163381 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.164496 13540 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:50.954722 17950 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:50.955243 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:50.918720 17654 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:51.012010 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 71395.3us
I20260807 08:01:51.017623 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.016693 17907 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.029372 13804 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:51.171986 13805 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:51.171332 13705 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:01:51.034518 13588 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:50.959790 17681 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20260807 08:01:50.924665 17836 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:50.992066 13657 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:51.002351 17920 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:50.939419 18014 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:51.100406 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:50.933192 17918 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.012422 18005 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:51.105718 13701 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
W20260807 08:01:51.106566 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 153295us
I20260807 08:01:51.167300 18016 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:51.117836 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.001870 17904 consensus_queue.cc:1046] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.341s
I20260807 08:01:51.028239 17242 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:51.130769 13587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:01:51.016106 13537 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.034981 17658 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.027181 18004 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
W20260807 08:01:51.214881 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 198821us
W20260807 08:01:51.138322 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 178269us
I20260807 08:01:51.215768 13769 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.111802 17983 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:51.143846 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 120406us
I20260807 08:01:51.144863 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.190614 17508 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.143226 17915 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.150195 17967 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:51.150130 17721 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:51.151502 17885 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.142910 17924 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:51.153005 17757 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:51.114133 17844 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.168459 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:01:51.269086 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 100659us
I20260807 08:01:51.269886 13540 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:51.014593 17723 consensus_queue.cc:1046] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.116s
I20260807 08:01:51.128844 17765 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.183398 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:51.283335 13708 raft_consensus.cc:1226] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:51.184417 17585 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:51.184598 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.184886 17943 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:51.180768 17907 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.187220 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.188819 17654 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.150264 17429 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.189314 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.170382 13711 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
W20260807 08:01:51.182307 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 190390us
I20260807 08:01:51.193215 13805 raft_consensus.cc:2396] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260807 08:01:51.195788 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:51.016868 17997 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:51.172086 17218 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:51.170306 17696 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:51.180935 17984 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:51.198446 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.173219 17499 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.173890 17449 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.219048 17985 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.222636 13706 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:51.333567 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.334262 13706 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:01:51.145309 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.225597 13710 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:51.227314 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.173954 17486 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.229233 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:51.229913 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:51.230254 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.231139 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.336262 13811 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:51.336632 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.336807 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.337142 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.337373 13814 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:51.337683 13811 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:51.231911 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.338142 13807 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 08:01:51.232683 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.232751 17242 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:51.233117 18029 log.cc:924] Time spent T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260807 08:01:51.234205 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.301798 17508 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:51.235057 17986 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.236021 17958 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.194831 18005 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.237025 18004 raft_consensus.cc:686] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:51.315361 17696 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.238579 17918 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.195166 17920 raft_consensus.cc:686] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:51.239434 17522 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.240965 17862 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.241714 18001 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.241736 17751 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:51.248350 13769 leader_election.cc:398] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:51.250378 17973 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.251006 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:51.364091 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 122784us
I20260807 08:01:51.229913 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.364308 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.364754 13804 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:51.364840 13769 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.365029 13710 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:51.257566 13580 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:51.260103 14580 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:51.367774 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 111834us
I20260807 08:01:51.368012 17909 raft_consensus.cc:2726] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:51.261592 13537 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:51.369688 17429 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260807 08:01:51.370074 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 108615us
I20260807 08:01:51.371201 13769 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.372032 13769 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.372386 13537 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.274716 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.373735 17924 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:01:51.277905 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.374143 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.237073 18012 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:51.375136 18012 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:01:51.375444 17967 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:01:51.375824 18012 raft_consensus.cc:686] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:51.375085 17924 consensus_queue.cc:1035] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260807 08:01:51.289148 13590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1771) took 3322 ms. Trace:
I20260807 08:01:51.381486 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.382117 13816 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:51.381783 13590 rpcz_store.cc:276] 0807 08:01:47.967111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:48.319179 (+352068us) service_pool.cc:227] Handling call
0807 08:01:48.360544 (+ 41365us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:50.824149 (+2463605us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:51.288867 (+464718us) spinlock_profiling.cc:243] Waited 2.37 s on lock 0x7b540021fcd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:51.289077 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5459345152}
I20260807 08:01:51.382298 17920 consensus_queue.cc:237] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.383145 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.291147 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.386179 14630 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 7: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:51.289831 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:51.387181 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 97446.6us
I20260807 08:01:51.297260 13540 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:51.302192 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.303056 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.304654 14622 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260807 08:01:51.348171 17696 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:51.310099 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:51.404747 14618 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:51.344681 17721 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.265779 17910 consensus_queue.cc:1046] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260807 08:01:51.419133 17910 raft_consensus.cc:3032] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:51.419525 17910 raft_consensus.cc:729] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:51.420439 17910 consensus_queue.cc:260] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.283650 17844 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:51.285051 17585 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.292613 17907 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.269378 17883 consensus_queue.cc:1035] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260807 08:01:51.338917 17242 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:51.436632 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.437258 13705 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:51.282851 17869 consensus_queue.cc:1035] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20260807 08:01:51.297933 17943 raft_consensus.cc:686] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:51.340245 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:51.340559 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 110733us
I20260807 08:01:51.381757 17486 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.446249 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.342237 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.309628 17997 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:01:51.343493 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.238461 18014 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.464164 13706 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:51.284770 17681 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.045s
I20260807 08:01:51.465512 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.291682 17938 consensus_queue.cc:1046] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260807 08:01:51.355291 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.467742 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.340727 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.252297 17916 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:51.376014 17967 raft_consensus.cc:686] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:51.254760 18013 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.361793 17751 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:51.379786 17985 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:51.278334 17885 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:51.364262 13703 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:51.394855 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.504377 13808 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:01:51.375203 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.396606 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.346655 18005 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:51.389753 13702 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:51.446269 17836 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:51.403510 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 106334us
I20260807 08:01:51.514844 13540 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.296766 13657 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:51.397729 13816 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:51.411353 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.411456 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.412750 13537 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.516731 13580 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:01:51.413503 17522 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:01:51.413793 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.413924 17958 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:51.347640 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.416304 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.414425 18004 consensus_queue.cc:237] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.415755 17990 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:51.376878 18012 consensus_queue.cc:237] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:51.421298 13769 connection.cc:232] RPC call timeout handler was delayed by -2.8195759430527687s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1545s for a response
I20260807 08:01:51.433372 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.424384 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:51.423897 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.438544 13661 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:51.530063 14629 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:51.439568 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.440376 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.441387 14583 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:51.301980 17765 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:51.345367 17508 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:51.468180 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.470166 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.425649 17721 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.437685 17585 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:51.474388 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:51.476574 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.436877 17897 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:51.439520 17909 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:51.571305 17909 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.572566 17658 raft_consensus.cc:3032] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:51.572944 17658 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:51.573690 17658 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:51.574441 18041 log.cc:924] Time spent T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20260807 08:01:51.574986 17658 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.446769 17844 raft_consensus.cc:686] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:51.401471 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.576637 13811 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:51.506657 14618 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:51.576903 17844 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.577978 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.578801 14618 raft_consensus.cc:2379] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:51.579566 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.434530 17973 consensus_queue.cc:1046] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260807 08:01:51.474247 17997 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:51.580713 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.582052 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.582854 14619 raft_consensus.cc:2379] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 6: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:51.474454 17910 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.583621 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.584748 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:51.520998 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 224328us
I20260807 08:01:51.521731 13710 raft_consensus.cc:2379] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:51.519601 13585 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:51.522905 14580 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:51.586627 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 63807.6us
I20260807 08:01:51.524080 13703 raft_consensus.cc:1261] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:51.521988 17958 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.589005 17654 consensus_queue.cc:1046] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:01:51.589344 17958 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:51.590246 17654 raft_consensus.cc:3032] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:51.590591 17654 raft_consensus.cc:729] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:51.591368 17654 consensus_queue.cc:260] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:51.593009 17654 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.503353 17751 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:51.593814 17740 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.593763 17897 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.594539 17740 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.595139 17740 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.502058 17915 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.531355 13588 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
W20260807 08:01:51.532228 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 93796.1us
I20260807 08:01:51.598642 13661 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:51.599145 17681 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:51.599521 17869 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:51.600092 17869 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.600206 13661 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:51.600679 17869 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:01:51.539636 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 98326.3us
W20260807 08:01:51.529173 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 116508us
I20260807 08:01:51.542116 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:51.542687 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:51.543820 13769 connection.cc:232] RPC call timeout handler was delayed by -2.790775943081826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
I20260807 08:01:51.546103 13590 raft_consensus.cc:2396] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260807 08:01:51.549429 13701 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 7: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:51.528999 14623 raft_consensus.cc:2379] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:51.518376 17885 raft_consensus.cc:686] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:51.563100 17508 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.543560 18014 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:51.570688 17904 raft_consensus.cc:3032] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:51.596771 17897 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.601470 13661 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:51.636683 14626 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
W20260807 08:01:51.640429 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 98384.8us
I20260807 08:01:51.527781 17990 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.617406 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.624222 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.612246 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.637229 13537 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.624658 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.646747 13816 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:51.602422 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:51.637887 13769 connection.cc:232] RPC call timeout handler was delayed by -2.790775943081826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
W20260807 08:01:51.647665 13769 connection.cc:232] RPC call timeout handler was delayed by -2.7070759430062026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.167s for a response
W20260807 08:01:51.648103 13769 connection.cc:232] RPC call timeout handler was delayed by -2.6872759431134909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
W20260807 08:01:51.648464 13769 connection.cc:232] RPC call timeout handler was delayed by -2.6296759429387748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20260807 08:01:51.648825 13769 connection.cc:232] RPC call timeout handler was delayed by -2.6296759429387748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
I20260807 08:01:51.644048 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:51.641126 13540 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.606824 13817 raft_consensus.cc:1226] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:51.528220 17603 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:51.638470 14619 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:51.546923 17765 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.538040 17522 raft_consensus.cc:686] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 8 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:51.638922 17904 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:51.614262 14580 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:51.671545 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 57369.3us
I20260807 08:01:51.651582 13710 raft_consensus.cc:2362] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8. Re-sending same reply.
I20260807 08:01:51.658190 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:51.658810 13811 raft_consensus.cc:2396] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:01:51.641934 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:51.649143 13769 connection.cc:232] RPC call timeout handler was delayed by -2.5693759429268539s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18230000000000002s for a response
W20260807 08:01:51.674163 13769 connection.cc:232] RPC call timeout handler was delayed by -2.4991759429685771s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
W20260807 08:01:51.674638 13769 connection.cc:232] RPC call timeout handler was delayed by -2.4595759429503232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
W20260807 08:01:51.674994 13769 connection.cc:232] RPC call timeout handler was delayed by -2.4595759429503232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
W20260807 08:01:51.675362 13769 connection.cc:232] RPC call timeout handler was delayed by -2.4415759430266917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260807 08:01:51.675706 13769 connection.cc:232] RPC call timeout handler was delayed by -2.4415759430266917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260807 08:01:51.676088 13769 connection.cc:232] RPC call timeout handler was delayed by -2.3380759430583566s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
I20260807 08:01:51.649559 17368 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:51.657464 17524 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:51.680992 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.629143 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.601972 17915 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.666774 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.606225 17986 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:51.667848 13537 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.663753 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:51.644703 17968 log.cc:924] Time spent T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20260807 08:01:51.659802 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.644873 17449 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election won for term 5
W20260807 08:01:51.676455 13769 connection.cc:232] RPC call timeout handler was delayed by -1.8997759430203587s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25670000000000004s for a response
W20260807 08:01:51.698164 13769 connection.cc:232] RPC call timeout handler was delayed by -1.8997759430203587s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25670000000000004s for a response
W20260807 08:01:51.698539 13769 connection.cc:232] RPC call timeout handler was delayed by -1.7721790510695428s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260807 08:01:51.650970 17990 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:51.650055 17508 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:51.551909 17721 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.679733 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.665290 13540 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:51.710301 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 45062.8us
I20260807 08:01:51.683405 17524 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.682436 13701 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:51.682997 17915 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.688992 14623 raft_consensus.cc:1226] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:51.717947 17524 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:51.676561 17909 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.667290 17916 raft_consensus.cc:2726] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:51.670205 13817 raft_consensus.cc:3032] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:51.732213 13817 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:51.732957 13817 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:51.733785 18036 log.cc:924] Time spent T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.386s user 0.004s sys 0.000s
I20260807 08:01:51.715217 17904 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.734918 17904 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.676460 17527 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.737705 17527 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.738106 17527 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.663251 17967 consensus_queue.cc:237] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:51.664085 18014 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260807 08:01:51.711156 13540 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:51.733911 13817 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:51.745378 13540 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.709245 13811 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:51.684072 14625 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:51.689304 17585 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:51.715272 17844 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.723994 17751 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:51.698400 17984 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.725752 17893 raft_consensus.cc:3032] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:51.639163 17943 consensus_queue.cc:237] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:51.726770 17740 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.727902 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.729182 17915 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.695904 13537 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.749987 14580 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.675868 17885 consensus_queue.cc:237] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.773294 13817 raft_consensus.cc:1261] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260807 08:01:51.773671 13817 pending_rounds.cc:76] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 4
I20260807 08:01:51.773877 13817 pending_rounds.cc:97] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260807 08:01:51.774112 13817 raft_consensus.cc:2866] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:51.775547 17909 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.776085 17909 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.776521 17909 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.754123 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:51.779075 17740 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:51.766080 13769 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:51.781418 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:51.781690 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 85895.1us
I20260807 08:01:51.782405 17757 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:51.782998 17757 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.783090 13769 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.783370 17757 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:51.783774 17757 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.784075 17916 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:51.784439 17916 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.784502 17658 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:51.784840 17916 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:51.785254 18055 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.077s user 0.000s sys 0.002s
I20260807 08:01:51.762444 17893 raft_consensus.cc:729] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:51.768306 13706 raft_consensus.cc:2379] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:51.689304 17242 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.822990 17242 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:51.715809 17508 raft_consensus.cc:686] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:51.714967 17449 raft_consensus.cc:686] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:51.788347 13809 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:51.788323 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:51.790740 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:51.827695 13811 raft_consensus.cc:2396] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260807 08:01:51.775568 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 121214us
I20260807 08:01:51.761047 17499 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:01:51.760908 18009 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:51.839185 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.840147 13579 raft_consensus.cc:2379] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:51.762537 17984 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:51.813472 18016 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:51.813715 17218 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:51.815274 17723 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:51.802037 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:51.847957 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 58694.9us
I20260807 08:01:51.852939 13540 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.850392 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.785112 17658 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.854478 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:51.789197 18013 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.872510 13589 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260807 08:01:51.850297 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 62065.6us
I20260807 08:01:51.858043 14583 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.860935 13585 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.879303 14583 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.862603 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:51.850081 17429 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:51.837002 17499 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.855285 17984 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.919979 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:51.921262 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:51.764428 17869 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.929333 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.757920 17522 consensus_queue.cc:237] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:51.932593 13587 raft_consensus.cc:595] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.854370 17218 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:51.937086 13585 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:51.886407 18013 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:51.882841 17938 raft_consensus.cc:3032] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:51.883026 17740 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:51.947088 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.883965 17985 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:51.937325 17218 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:51.882723 18016 raft_consensus.cc:686] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:51.953272 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:51.969571 13711 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 8: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:51.922622 17499 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:51.970085 13701 raft_consensus.cc:2379] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:51.971158 13769 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.968663 13708 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:51.924213 18014 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260807 08:01:51.971843 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 118981us
I20260807 08:01:51.866494 17723 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.982765 13540 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.976222 14629 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:01:51.984114 13709 raft_consensus.cc:2396] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:01:51.881758 17990 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:51.955683 18013 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:51.990082 13540 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:51.991860 13769 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:51.949033 17429 raft_consensus.cc:686] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:51.969235 18001 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:51.994143 13540 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:52.003971 13540 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.005148 13701 raft_consensus.cc:1261] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 08:01:51.962320 17765 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:51.953715 17938 raft_consensus.cc:729] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:51.961623 17449 consensus_queue.cc:237] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:51.996986 17585 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:51.998386 18052 log.cc:924] Time spent T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.243s user 0.001s sys 0.000s
I20260807 08:01:51.998941 18014 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:51.969008 18012 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 08:01:52.012271 18058 log.cc:924] Time spent T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
W20260807 08:01:52.014194 18063 log.cc:924] Time spent T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260807 08:01:52.017156 18016 consensus_queue.cc:237] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.020838 17990 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:51.934149 17950 consensus_queue.cc:1046] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260807 08:01:52.026715 17658 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:52.034974 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.155730 13708 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:01:52.055315 18009 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:52.036881 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.036778 17757 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:52.164796 18060 log.cc:924] Time spent T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.457s user 0.002s sys 0.002s
I20260807 08:01:52.165802 17757 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:52.167096 14620 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:55594
I20260807 08:01:52.167801 14620 raft_consensus.cc:595] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:52.168226 14620 raft_consensus.cc:603] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:52.038339 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.037957 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.038793 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.169932 13703 raft_consensus.cc:2354] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5.
I20260807 08:01:51.986061 17967 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:51.885507 17893 consensus_queue.cc:260] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.073098 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:52.046025 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:52.072206 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:52.174005 14625 raft_consensus.cc:1226] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:52.174572 14629 raft_consensus.cc:2379] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:52.073066 14583 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:52.175405 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 102412us
W20260807 08:01:52.056780 17696 consensus_meta.cc:287] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.624s user 0.006s sys 0.006s
I20260807 08:01:52.176088 17696 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.176908 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:01:52.042572 18005 consensus_meta.cc:287] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.519s user 0.007s sys 0.004s
I20260807 08:01:52.177759 18005 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:52.178524 17696 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:52.048216 17869 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:52.110762 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 446) took 1158 ms. Trace:
I20260807 08:01:52.046371 17751 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:51.988396 17924 consensus_queue.cc:1035] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.110s
I20260807 08:01:52.206012 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.115381 17429 consensus_queue.cc:237] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.123589 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.146924 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:52.144363 13661 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:52.214143 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 69911.2us
I20260807 08:01:52.055899 17765 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:51.959261 17508 consensus_queue.cc:237] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.151177 17938 consensus_queue.cc:260] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.156958 13708 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 08:01:52.035418 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.231024 13705 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:52.096721 17872 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:52.235301 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.235958 13809 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:52.163938 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.238793 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.239758 13805 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:01:52.209096 18052 log.cc:924] Time spent T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260807 08:01:52.108850 17986 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:52.185056 13709 rpcz_store.cc:276] 0807 08:01:50.952811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:51.469925 (+517114us) service_pool.cc:227] Handling call
0807 08:01:52.110572 (+640647us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b5400292a54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:52.110706 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1011363200}
W20260807 08:01:52.186101 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 182210us
W20260807 08:01:52.156296 17997 consensus_meta.cc:287] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.575s user 0.005s sys 0.007s
I20260807 08:01:52.187141 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:52.198134 13580 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:01:52.156622 18012 raft_consensus.cc:2741] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260807 08:01:52.223961 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.089560 17527 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.215556 13661 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:52.268894 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 277074us
W20260807 08:01:52.201819 17654 consensus_meta.cc:287] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.548s user 0.017s sys 0.003s
I20260807 08:01:52.161791 17916 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:52.168133 18001 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:52.240834 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.185091 17983 raft_consensus.cc:2726] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:52.307075 13816 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 8
I20260807 08:01:52.243918 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:52.213447 18063 log.cc:924] Time spent T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260807 08:01:52.210172 17934 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:52.249099 13540 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:52.263547 17967 raft_consensus.cc:3032] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:52.263671 18013 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.259946 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:52.222514 17765 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:52.228832 17585 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
W20260807 08:01:52.269076 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 53604.9us
I20260807 08:01:52.296550 17883 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:01:52.232797 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.257778 14623 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260807 08:01:52.259063 17986 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:52.310918 13809 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:52.199306 17985 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.060s
W20260807 08:01:52.296074 18055 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260807 08:01:52.321537 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:52.270555 13769 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.273629 17938 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:52.329717 13661 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:52.266712 17997 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.319159 17967 raft_consensus.cc:729] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:01:52.389365 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 59724.7us
I20260807 08:01:52.390365 13661 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:52.332127 13808 raft_consensus.cc:3037] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 5
I20260807 08:01:52.312386 18001 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:52.249193 17915 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:52.341353 17524 consensus_meta.cc:287] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.611s user 0.006s sys 0.004s
I20260807 08:01:52.312388 17991 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:52.310910 18005 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:52.326794 18044 log.cc:924] Time spent T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
W20260807 08:01:52.345614 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 142272us
I20260807 08:01:52.348727 13701 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:52.325187 17765 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:52.330780 14626 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:01:52.387233 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 143432us
I20260807 08:01:52.327720 17934 raft_consensus.cc:686] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:52.334829 14620 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:55594
I20260807 08:01:52.346306 17585 raft_consensus.cc:686] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:52.402299 14619 raft_consensus.cc:1226] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:52.395097 17654 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 08:01:52.404206 17958 consensus_meta.cc:287] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.814s user 0.010s sys 0.003s
W20260807 08:01:52.410177 17844 consensus_meta.cc:287] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.649s user 0.012s sys 0.006s
W20260807 08:01:52.409180 18068 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.177s user 0.000s sys 0.001s
I20260807 08:01:52.351739 17985 consensus_queue.cc:1225] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer 77da774f2bb54cf28302d53d0db224ad log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260807 08:01:52.417002 14630 raft_consensus.cc:2396] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260807 08:01:52.405076 17907 consensus_meta.cc:287] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.956s user 0.010s sys 0.003s
I20260807 08:01:52.436334 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.470547 14583 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.339190 17883 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:52.444635 17909 consensus_meta.cc:287] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.668s user 0.010s sys 0.002s
I20260807 08:01:52.399567 17910 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:52.445914 13661 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:52.437342 17654 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:52.393221 17723 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:52.451013 18078 log.cc:924] Time spent T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.326s user 0.002s sys 0.000s
I20260807 08:01:52.477674 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:52.406236 17524 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:52.449357 17844 raft_consensus.cc:2726] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:52.478569 13579 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260807 08:01:52.454685 14620 raft_consensus.cc:595] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:52.511431 17721 consensus_meta.cc:287] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.789s user 0.011s sys 0.001s
I20260807 08:01:52.464056 17983 consensus_queue.cc:1035] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
W20260807 08:01:52.511864 18100 log.cc:924] Time spent T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
W20260807 08:01:52.514219 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 274377us
I20260807 08:01:52.543120 13657 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:52.515120 17934 consensus_queue.cc:237] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.468510 17585 consensus_queue.cc:237] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:52.507735 18083 log.cc:924] Time spent T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.242s user 0.000s sys 0.001s
I20260807 08:01:52.517874 13584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 599) took 1586 ms. Trace:
I20260807 08:01:52.558398 13584 rpcz_store.cc:276] 0807 08:01:50.930969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:51.005918 (+ 74949us) service_pool.cc:227] Handling call
0807 08:01:51.025411 (+ 19493us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:51.025865 (+ 454us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:51.025892 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:51.656535 (+630643us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b2c00025b18. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1b94ee11 00007f6b1b95e442 00007f6b1cfa6c11 00007f6b1cf6b58a 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772
0807 08:01:51.665690 (+ 9155us) log.cc:842] Serialized 26 byte log entry
0807 08:01:51.960490 (+294800us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:51.960541 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:51.960920 (+ 379us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:52.517587 (+556667us) raft_consensus.cc:1671] finished
0807 08:01:52.517606 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:52.517816 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":72114,"spinlock_wait_cycles":6673664,"thread_start_us":269203,"threads_started":1,"wal-append.queue_time_us":450087}
W20260807 08:01:52.474453 17904 consensus_meta.cc:287] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.738s user 0.026s sys 0.008s
I20260807 08:01:52.520359 17603 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:52.561719 17869 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:52.562384 17869 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:52.562988 17869 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:52.566457 17997 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:52.567432 17997 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:52.568830 18113 raft_consensus.cc:2726] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:52.573603 13589 raft_consensus.cc:2379] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:52.475927 17883 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:52.529881 13816 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:01:52.477234 17654 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:52.475407 17910 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:52.448227 17958 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:52.544202 18091 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20260807 08:01:52.520293 17617 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:52.584161 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.527319 14620 raft_consensus.cc:603] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:01:52.556114 18076 consensus_queue.cc:1046] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:52.451997 17967 consensus_queue.cc:260] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.608172 13592 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:52.471729 17907 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.477707 17909 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:52.594653 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 124137us
I20260807 08:01:52.574877 17916 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:52.591849 17910 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:52.545747 17527 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:52.655200 17869 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.542152 17721 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.684607 14627 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:52.600720 18013 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:52.709853 14618 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:52.714422 17869 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:52.708803 17915 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:52.754235 17524 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:52.765919 17909 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:52.770964 17721 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:52.782305 17885 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:52.783051 17885 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:52.783668 17885 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:01:52.793344 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 315739us
I20260807 08:01:52.793766 14583 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:52.794732 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 524199us
I20260807 08:01:52.796025 13769 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:52.796792 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.797605 13816 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 7
I20260807 08:01:52.798715 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:52.799505 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 550450us
W20260807 08:01:52.769280 17984 consensus_meta.cc:287] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.821s user 0.011s sys 0.001s
I20260807 08:01:52.804008 13808 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:52.804286 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 588) took 1394 ms. Trace:
I20260807 08:01:52.804668 14630 rpcz_store.cc:276] 0807 08:01:51.409958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:51.421991 (+ 12033us) service_pool.cc:227] Handling call
0807 08:01:52.804101 (+1382110us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b54003309d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:52.804231 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2230142208}
I20260807 08:01:52.808583 13816 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:52.810065 13769 leader_election.cc:398] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5.
I20260807 08:01:52.813796 13769 leader_election.cc:401] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260807 08:01:52.817127 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.818176 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.818902 13809 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 6
I20260807 08:01:52.820236 13814 raft_consensus.cc:1226] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:52.819164 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.801749 17967 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:52.812194 18124 log.cc:924] Time spent T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260807 08:01:52.808805 17603 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:52.823418 13811 raft_consensus.cc:2379] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:52.824384 14580 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:52.809450 17721 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:52.834371 13769 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.846210 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:52.846840 14630 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:52.848124 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:52.848899 14629 raft_consensus.cc:3037] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:52.854595 17924 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:01:52.855121 17924 raft_consensus.cc:686] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:52.844321 18094 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5.
I20260807 08:01:52.812288 17958 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:52.863703 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 63609.5us
W20260807 08:01:52.868206 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 271353us
I20260807 08:01:52.809614 18016 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:52.873906 13537 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:52.826040 17973 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:52.858894 13809 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:52.822427 18013 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:52.822810 18105 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:52.879208 13430 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56582 (request call id 306) took 12961 ms (client timeout 15000 ms). Trace:
I20260807 08:01:52.813861 17984 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:52.812407 18012 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260807 08:01:52.844333 17617 raft_consensus.cc:686] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 7 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:52.879886 14630 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:52.875595 13807 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:52.882421 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:52.875972 14629 raft_consensus.cc:2454] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:52.887425 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:52.874025 18107 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:52.888530 18107 raft_consensus.cc:686] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:52.888571 13769 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:52.890020 18122 log.cc:924] Time spent T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.202s user 0.000s sys 0.001s
I20260807 08:01:52.889573 18107 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:52.880831 13430 rpcz_store.cc:269] 0807 08:01:39.916434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:39.920553 (+ 4119us) service_pool.cc:227] Handling call
0807 08:01:52.874551 (+12953998us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:01:47.843341 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:01:48.080607 (+237266us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:01:48.092760 (+ 12153us) write_op.cc:435] Acquired schema lock
0807 08:01:48.116911 (+ 24151us) tablet.cc:660] Decoding operations
0807 08:01:48.714135 (+597224us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:01:48.764547 (+ 50412us) write_op.cc:641] Partition lock acquired for write op
0807 08:01:48.766829 (+ 2282us) tablet.cc:683] Acquiring locks for 11 operations
0807 08:01:48.882586 (+115757us) tablet.cc:699] Row locks acquired
0807 08:01:48.882636 (+ 50us) write_op.cc:260] PREPARE: finished
0807 08:01:48.882860 (+ 224us) write_op.cc:270] Start()
0807 08:01:48.883058 (+ 198us) write_op.cc:276] Timestamp: P: 1786089708882782 usec, L: 0
0807 08:01:48.883103 (+ 45us) op_driver.cc:331] REPLICATION: starting
0807 08:01:48.884014 (+ 911us) log.cc:842] Serialized 4402 byte log entry
0807 08:01:50.564930 (+1680916us) op_driver.cc:444] REPLICATION: finished
0807 08:01:51.369433 (+804503us) write_op.cc:301] APPLY: starting
0807 08:01:51.369645 (+ 212us) tablet.cc:1365] starting BulkCheckPresence
0807 08:01:51.369926 (+ 281us) tablet.cc:1368] finished BulkCheckPresence
0807 08:01:51.370061 (+ 135us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:01:51.371377 (+ 1316us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:01:51.371769 (+ 392us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0807 08:01:51.371815 (+ 46us) write_op.cc:312] APPLY: finished
0807 08:01:51.372150 (+ 335us) log.cc:842] Serialized 83 byte log entry
0807 08:01:51.372309 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:51.372442 (+ 133us) write_op.cc:454] Released schema lock
0807 08:01:51.372513 (+ 71us) write_op.cc:341] FINISH: Updating metrics
0807 08:01:51.418388 (+ 45875us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:01:51.418442 (+ 54us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":325638,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":751604,"apply.run_cpu_time_us":4075,"apply.run_wall_time_us":59511,"num_ops":11,"prepare.queue_time_us":150960,"prepare.run_cpu_time_us":18116,"prepare.run_wall_time_us":1358443,"raft.queue_time_us":178387,"raft.run_cpu_time_us":38684,"raft.run_wall_time_us":709431,"replication_time_us":1675744,"thread_start_us":826237,"threads_started":4,"wal-append.queue_time_us":568485,"wal-append.run_cpu_time_us":99655,"wal-append.run_wall_time_us":2880434}]]}
I20260807 08:01:52.824581 17915 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
W20260807 08:01:52.893256 18117 log.cc:924] Time spent T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.263s user 0.001s sys 0.000s
I20260807 08:01:52.889585 17429 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:52.898362 17429 raft_consensus.cc:686] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:01:52.899168 17499 consensus_meta.cc:287] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.911s user 0.012s sys 0.002s
I20260807 08:01:52.899657 17499 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:52.876214 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 55095.6us
I20260807 08:01:52.899276 17429 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.906756 13540 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.918613 14580 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:52.918825 17740 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:52.919474 17740 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.919638 17617 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.920485 17643 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:52.921017 17643 raft_consensus.cc:686] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:52.921813 17643 consensus_queue.cc:237] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:52.922722 14627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 591) took 1499 ms (client timeout 1941 ms). Trace:
W20260807 08:01:52.923460 17218 consensus_meta.cc:287] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.937s user 0.012s sys 0.016s
W20260807 08:01:52.923130 14627 rpcz_store.cc:269] 0807 08:01:51.423368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:51.423606 (+ 238us) service_pool.cc:227] Handling call
0807 08:01:52.922529 (+1498923us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540020dad4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:52.922660 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2654404352}
I20260807 08:01:52.923904 17218 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.924376 17958 raft_consensus.cc:2781] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:52.924275 18014 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:52.901465 17757 raft_consensus.cc:2781] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:52.913738 13661 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:52.912525 13657 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:52.928843 13540 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:52.929682 13537 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:52.876778 17986 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:52.932487 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:52.936885 14627 raft_consensus.cc:1226] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:52.936872 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:52.938521 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.934116 18009 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:52.885812 18013 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.882953 17973 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.941160 17973 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:52.867820 17924 consensus_queue.cc:237] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.902351 18105 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.947070 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:52.948611 14583 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.948977 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:52.949827 14627 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:52.882586 17765 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:52.951679 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:52.951813 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:52.952471 13816 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 8: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:52.954154 13590 raft_consensus.cc:1226] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 7. Current term is 8. Ops: []
I20260807 08:01:52.954632 14583 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.958904 13540 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.917171 17907 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:52.973503 18005 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:52.974010 17499 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:52.952916 17757 raft_consensus.cc:686] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:52.960858 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:52.975085 14625 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:52.975785 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:52.975449 17757 consensus_queue.cc:237] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:52.976202 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:52.976863 14625 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:52.977160 14626 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 08:01:52.960556 18013 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:52.944842 18014 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:52.978152 14618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 592) took 1542 ms (client timeout 1633 ms). Trace:
I20260807 08:01:52.978276 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:52.978902 14622 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:52.978643 14618 rpcz_store.cc:269] 0807 08:01:51.435995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:51.579315 (+143320us) service_pool.cc:227] Handling call
0807 08:01:52.977964 (+1398649us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54006045d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:52.978095 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2597389568}
I20260807 08:01:52.979629 18014 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:52.952621 17938 consensus_meta.cc:287] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.554s user 0.011s sys 0.000s
I20260807 08:01:52.952587 14630 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:52.980950 13592 pending_rounds.cc:76] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Aborting all ops after (but not including) 6
I20260807 08:01:52.961138 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:52.981294 13592 pending_rounds.cc:97] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260807 08:01:52.981626 13592 raft_consensus.cc:2866] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:52.962725 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:52.965201 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:52.968820 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:52.969103 17740 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:52.969893 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:52.970041 17765 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:52.970963 13814 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 08:01:52.971398 18118 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:52.971812 17950 consensus_queue.cc:1046] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:01:52.960071 17844 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:52.953259 18076 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:52.944458 18105 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:52.987874 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:52.934798 18119 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260807 08:01:52.988664 14629 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
W20260807 08:01:52.991220 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 79617.9us
I20260807 08:01:52.984571 13580 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:01:52.999224 13540 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:52.999871 14622 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:52.875769 18127 consensus_queue.cc:1046] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.117s
I20260807 08:01:53.009603 14583 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:53.048463 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.049824 13711 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:53.002305 17524 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:53.014540 14630 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 08:01:52.945595 17958 raft_consensus.cc:686] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:52.982225 17916 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:53.039431 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:53.062765 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:53.016999 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:53.017762 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.064554 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.046432 13811 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:01:53.065577 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 56101us
W20260807 08:01:53.066227 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 67078.4us
I20260807 08:01:53.047421 13706 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 08:01:53.067844 13540 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:52.982084 18005 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:52.934414 17522 consensus_queue.cc:1046] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20260807 08:01:53.057765 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.059278 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.070689 13708 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:53.063088 13814 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:53.008580 17915 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:01:53.072860 13709 raft_consensus.cc:2396] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 08:01:53.020522 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.075084 13710 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:53.002269 18113 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:53.079125 13705 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 7: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:53.080340 14583 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:53.081885 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:53.087503 13706 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:53.087985 18014 consensus_queue.cc:1046] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 08:01:53.088935 17934 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:53.089164 18090 raft_consensus.cc:3032] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:53.089773 18090 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:53.076152 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:52.979703 17654 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:53.090811 18090 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:53.091837 13710 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:01:53.092303 18090 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:53.093469 17654 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:53.094858 18009 raft_consensus.cc:3032] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:53.095515 18009 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:53.098376 18009 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:53.102010 13540 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:53.103477 17524 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:53.106325 17524 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:53.040086 17844 raft_consensus.cc:686] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:53.129279 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 144431us
I20260807 08:01:53.047935 18076 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:53.128377 13580 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:53.069697 18013 raft_consensus.cc:3032] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:53.008793 18004 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260807 08:01:53.073052 18005 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:53.149049 13700 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:53.082530 17920 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:53.162540 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:53.168668 13807 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:53.084249 17841 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:53.117063 17508 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:53.180289 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:53.073918 17499 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.088s
I20260807 08:01:53.206142 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:53.161954 18013 raft_consensus.cc:729] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:53.161561 17924 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:53.192211 18083 log.cc:924] Time spent T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260807 08:01:53.214344 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:53.179107 18009 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:53.219105 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:53.234017 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 185066us
I20260807 08:01:53.190797 17920 raft_consensus.cc:686] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:53.239400 13657 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:53.231043 13708 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:53.233443 13811 raft_consensus.cc:1261] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260807 08:01:53.240738 13584 raft_consensus.cc:2354] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:53.204381 17429 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:53.252000 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:53.252202 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:53.253422 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.254419 13709 raft_consensus.cc:2379] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:53.245007 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:53.245637 13657 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:53.246661 13710 raft_consensus.cc:1261] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:01:53.247278 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.248106 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.259361 13540 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:53.249261 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:53.209117 17986 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:53.251050 13580 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:53.259380 13706 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:53.231457 17984 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:53.261088 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:53.261981 13769 leader_election.cc:398] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:53.274462 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:53.274894 13769 leader_election.cc:401] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260807 08:01:53.263537 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:53.277801 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.278026 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.264443 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:53.278740 13710 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 8: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:53.260535 13809 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:53.270427 13816 raft_consensus.cc:3032] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:53.280622 13816 raft_consensus.cc:729] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:53.282009 13769 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
W20260807 08:01:53.267951 17883 consensus_meta.cc:287] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.665s user 0.007s sys 0.004s
W20260807 08:01:53.262025 18001 consensus_meta.cc:287] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.841s user 0.002s sys 0.011s
I20260807 08:01:53.298725 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:53.283790 13705 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260807 08:01:53.284233 13709 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 08:01:53.269958 14623 raft_consensus.cc:1226] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:53.301595 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:53.287274 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.285126 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:53.167542 18076 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:53.281451 13816 consensus_queue.cc:260] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:53.355880 13816 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
W20260807 08:01:53.289932 18149 log.cc:924] Time spent T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260807 08:01:53.222066 17522 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.051s
I20260807 08:01:53.257789 17218 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:53.274559 17910 consensus_meta.cc:287] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.585s user 0.010s sys 0.002s
W20260807 08:01:53.294072 18141 log.cc:924] Time spent T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20260807 08:01:53.293960 17920 consensus_queue.cc:237] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:53.327142 13708 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:53.248899 17524 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260807 08:01:53.244388 17844 consensus_queue.cc:237] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:53.346472 17885 consensus_meta.cc:287] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.562s user 0.014s sys 0.000s
I20260807 08:01:53.236434 17924 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:53.298344 17883 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:53.327466 13710 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:01:53.348810 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:53.350896 13711 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:53.352622 13706 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:01:53.270351 17967 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:53.308637 17617 raft_consensus.cc:2781] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:01:53.271098 17508 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:53.377662 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.440001 13705 raft_consensus.cc:2379] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 5: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:53.394435 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:53.442194 13587 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:53.208616 17958 consensus_queue.cc:237] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:53.412595 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 369223us
I20260807 08:01:53.419250 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.370049 13809 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:53.429623 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:53.363847 18012 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:53.456914 13816 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260807 08:01:53.468763 13708 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:53.469671 13537 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:53.266405 17429 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:53.364369 18152 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:01:53.341205 18001 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:53.284852 18013 consensus_queue.cc:260] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:53.408185 17910 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:53.416002 17218 consensus_queue.cc:1046] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.142s
W20260807 08:01:53.896176 14627 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.021s sys 0.058s
W20260807 08:01:53.581768 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 227575us
I20260807 08:01:53.504072 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.947553 13816 pending_rounds.cc:76] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 5
I20260807 08:01:53.947964 13816 pending_rounds.cc:97] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260807 08:01:53.948278 13816 raft_consensus.cc:2866] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:53.471838 13700 raft_consensus.cc:2379] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 9: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
W20260807 08:01:53.589787 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 307852us
W20260807 08:01:53.942266 14627 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.961s user 0.023s sys 0.059s
I20260807 08:01:53.566838 13814 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:53.549254 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:53.941354 13657 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.009s sys 0.007s
W20260807 08:01:54.028633 13661 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.009s sys 0.007s
W20260807 08:01:54.029096 13661 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.568s user 0.009s sys 0.007s
I20260807 08:01:53.548568 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:54.011049 13657 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.578s user 0.010s sys 0.008s
W20260807 08:01:53.917798 18176 log.cc:924] Time spent T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
I20260807 08:01:54.067766 13588 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:54.068060 13580 raft_consensus.cc:2379] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 7: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
W20260807 08:01:54.068502 13817 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.000s sys 0.010s
W20260807 08:01:54.068827 13817 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s user 0.000s sys 0.010s
W20260807 08:01:53.798393 18146 log.cc:924] Time spent T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.540s user 0.000s sys 0.001s
W20260807 08:01:53.769573 18144 log.cc:924] Time spent T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.469s user 0.001s sys 0.000s
W20260807 08:01:53.588570 18127 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.509s user 0.023s sys 0.102s
I20260807 08:01:54.000535 13540 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:54.029412 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 568181us
I20260807 08:01:53.918722 17883 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:53.925129 17924 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:53.941741 18012 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:54.073575 14580 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.006s sys 0.006s
W20260807 08:01:54.115721 14580 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.006s sys 0.006s
I20260807 08:01:54.112283 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:54.116122 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 613205us
I20260807 08:01:53.425920 17885 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:53.547878 17508 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.084043 13711 raft_consensus.cc:1226] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 08:01:54.001495 18016 raft_consensus.cc:2726] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:54.086637 13708 raft_consensus.cc:1226] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:53.915154 18076 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:54.087711 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:53.635178 14623 raft_consensus.cc:1261] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260807 08:01:54.089926 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:53.708941 17524 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:53.622637 17986 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.133950 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:54.134676 13706 raft_consensus.cc:2354] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:53.444741 17967 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.134836 13711 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:54.095510 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:54.132012 13587 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:54.136138 13710 raft_consensus.cc:2379] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 7: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:54.096154 13709 raft_consensus.cc:1226] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:54.041797 18181 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:54.089879 17585 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s user 0.009s sys 0.013s
I20260807 08:01:54.038678 18152 raft_consensus.cc:686] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:54.140486 18152 consensus_queue.cc:237] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.143640 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:54.133358 17508 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.147759 18001 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:54.148854 17508 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:54.150169 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 611) took 2315 ms. Trace:
I20260807 08:01:54.121763 18012 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:54.150525 13579 rpcz_store.cc:276] 0807 08:01:51.835064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:51.908712 (+ 73648us) service_pool.cc:227] Handling call
0807 08:01:51.910858 (+ 2146us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:52.520629 (+609771us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.149909 (+1629280us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b54002053d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:54.150114 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1304987904}
I20260807 08:01:54.151896 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:54.152657 13579 raft_consensus.cc:2379] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 9: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:54.153072 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1791) took 1023 ms. Trace:
I20260807 08:01:54.154211 17524 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.134186 14623 pending_rounds.cc:76] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Aborting all ops after (but not including) 6
I20260807 08:01:54.161980 14623 pending_rounds.cc:97] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260807 08:01:54.162422 14623 raft_consensus.cc:2866] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:54.163241 17910 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:54.164860 13708 raft_consensus.cc:1226] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260807 08:01:54.167567 17934 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260807 08:01:54.168659 18090 consensus_meta.cc:287] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.076s user 0.009s sys 0.007s
I20260807 08:01:54.169178 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:54.138291 18181 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.171571 17950 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.174842 18014 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 08:01:54.177666 13701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1802) took 2143 ms (client timeout 1578 ms). Trace:
W20260807 08:01:54.178119 13701 rpcz_store.cc:269] 0807 08:01:52.033865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.088525 (+ 54660us) service_pool.cc:227] Handling call
0807 08:01:54.177456 (+2088931us) spinlock_profiling.cc:243] Waited 83.4 ms on lock 0x7b5400408154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:54.177571 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191735552}
I20260807 08:01:54.130041 17924 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.181108 17893 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:54.181650 17924 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.182804 18181 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:54.153347 13579 rpcz_store.cc:276] 0807 08:01:53.129971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.151624 (+1021653us) service_pool.cc:227] Handling call
0807 08:01:54.153025 (+ 1401us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:54.193712 13579 raft_consensus.cc:1226] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8. Ops: []
I20260807 08:01:54.194113 13587 raft_consensus.cc:1226] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8. Ops: []
I20260807 08:01:54.437953 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1996) took 1044 ms. Trace:
I20260807 08:01:54.438558 13817 rpcz_store.cc:276] 0807 08:01:53.393637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:53.393759 (+ 122us) service_pool.cc:227] Handling call
0807 08:01:53.404181 (+ 10422us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:53.407181 (+ 3000us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 08:01:53.436728 (+ 29547us) log.cc:842] Serialized 14 byte log entry
0807 08:01:54.437492 (+1000764us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:54.437592 (+ 100us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:54.437652 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.437744 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:54.437884 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29765,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":10970,"thread_start_us":1000300,"threads_started":2,"wal-append.queue_time_us":739270}
W20260807 08:01:54.424161 13589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1908) took 2238 ms (client timeout 2043 ms). Trace:
W20260807 08:01:54.444448 13589 rpcz_store.cc:269] 0807 08:01:52.186487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.186839 (+ 352us) service_pool.cc:227] Handling call
0807 08:01:54.423895 (+2237056us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b540025dd54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:54.424065 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":673560320}
I20260807 08:01:54.448539 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1798) took 1308 ms. Trace:
I20260807 08:01:54.449107 13579 rpcz_store.cc:276] 0807 08:01:53.140001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.185210 (+1045209us) service_pool.cc:227] Handling call
0807 08:01:54.192955 (+ 7745us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:54.448117 (+255162us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.448462 (+ 345us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:54.157372 17524 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:54.460693 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 450381us
I20260807 08:01:54.471444 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:54.473735 13769 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:54.481882 13589 raft_consensus.cc:1226] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260807 08:01:54.515393 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1802) took 1339 ms. Trace:
I20260807 08:01:54.515980 13589 rpcz_store.cc:276] 0807 08:01:53.175837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.481118 (+1305281us) service_pool.cc:227] Handling call
0807 08:01:54.481610 (+ 492us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:54.515035 (+ 33425us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.515323 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:54.517666 13589 raft_consensus.cc:1226] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 8. Current term is 9. Ops: []
W20260807 08:01:54.445453 17617 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.852s user 0.055s sys 0.068s
I20260807 08:01:54.149435 17508 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.368888 17585 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.573750 13580 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:54.586617 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1806) took 1395 ms. Trace:
I20260807 08:01:54.185621 18090 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:54.437307 17997 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.377375 17885 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:54.595656 13589 rpcz_store.cc:276] 0807 08:01:53.191130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.517022 (+1325892us) service_pool.cc:227] Handling call
0807 08:01:54.517424 (+ 402us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:54.574327 (+ 56903us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.586532 (+ 12205us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:54.573112 18005 consensus_meta.cc:287] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.382s user 0.012s sys 0.002s
I20260807 08:01:54.437899 17924 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:54.605602 13589 raft_consensus.cc:1226] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:54.172375 17950 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:54.595199 17617 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.078s user 0.055s sys 0.069s
I20260807 08:01:54.184672 17986 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.218371 18200 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.389s
I20260807 08:01:54.611110 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1807) took 1419 ms. Trace:
W20260807 08:01:54.608634 18113 consensus_meta.cc:287] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 1.483s user 0.005s sys 0.010s
I20260807 08:01:54.459344 18210 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:54.624222 18009 consensus_meta.cc:287] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 1.377s user 0.009s sys 0.006s
I20260807 08:01:54.638226 13589 rpcz_store.cc:276] 0807 08:01:53.191445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.604931 (+1413486us) service_pool.cc:227] Handling call
0807 08:01:54.605346 (+ 415us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:54.610758 (+ 5412us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.611037 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:54.607545 18090 raft_consensus.cc:686] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:54.607149 17429 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:54.573586 13767 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.013s sys 0.026s
I20260807 08:01:54.645169 13584 raft_consensus.cc:2354] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 7.
I20260807 08:01:54.625257 18196 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:54.643787 18210 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.650597 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1788) took 1523 ms. Trace:
I20260807 08:01:54.649947 18196 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.651051 13587 rpcz_store.cc:276] 0807 08:01:53.127174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.170384 (+1043210us) service_pool.cc:227] Handling call
0807 08:01:54.170835 (+ 451us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:54.453017 (+282182us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.650271 (+197254us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b54001365d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:54.650524 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52770560}
I20260807 08:01:54.652542 18210 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:54.655189 17967 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:54.610478 18005 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.658988 17950 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:54.607856 17997 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.641546 18113 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:54.633491 17883 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:54.660804 13579 raft_consensus.cc:1226] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260807 08:01:54.615617 17617 raft_consensus.cc:686] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:54.661167 17883 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.663635 18154 raft_consensus.cc:3032] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:54.663054 17617 consensus_queue.cc:237] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.664175 18154 raft_consensus.cc:729] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:54.666769 17986 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:54.668318 17997 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:54.601617 17508 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.669405 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 623) took 1742 ms. Trace:
I20260807 08:01:54.669032 18090 consensus_queue.cc:237] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.669847 13592 rpcz_store.cc:276] 0807 08:01:52.927502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.927656 (+ 154us) service_pool.cc:227] Handling call
0807 08:01:52.933116 (+ 5460us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:53.002515 (+ 69399us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:53.002543 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:53.003062 (+ 519us) log.cc:842] Serialized 52 byte log entry
0807 08:01:53.417539 (+414477us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:53.417594 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.668960 (+1251366us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b54002058d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:54.669008 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:54.669062 (+ 54us) raft_consensus.cc:1671] finished
0807 08:01:54.669075 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:54.669334 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11679,"raft.queue_time_us":8522,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":3065,"spinlock_wait_cycles":109528832,"thread_start_us":344597,"threads_started":1,"wal-append.queue_time_us":1059932}
I20260807 08:01:54.638902 18206 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:54.671713 13703 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:54.633994 18211 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:54.672389 17973 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.672851 18211 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.673346 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1803) took 1497 ms. Trace:
W20260807 08:01:54.673775 13703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1812) took 2450 ms (client timeout 1548 ms). Trace:
I20260807 08:01:54.673316 18211 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:54.677742 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:54.674402 13703 rpcz_store.cc:269] 0807 08:01:52.223505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.223655 (+ 150us) service_pool.cc:227] Handling call
0807 08:01:54.673597 (+2449942us) spinlock_profiling.cc:243] Waited 2.41 s on lock 0x7b54002ff154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:54.673694 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5532143360}
I20260807 08:01:54.674057 13579 rpcz_store.cc:276] 0807 08:01:53.176244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.660056 (+1483812us) service_pool.cc:227] Handling call
0807 08:01:54.660545 (+ 489us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:54.661366 (+ 821us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.673285 (+ 11919us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 08:01:54.674510 13767 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.014s sys 0.026s
I20260807 08:01:54.679191 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:54.683312 13816 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 08:01:54.678264 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:54.693861 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:54.684417 13535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.014s sys 0.013s
I20260807 08:01:54.699604 13708 raft_consensus.cc:1226] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 4. Current term is 5. Ops: []
W20260807 08:01:54.699741 13535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.015s sys 0.013s
I20260807 08:01:54.695523 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:54.601933 17585 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.653385 18196 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:54.704340 17986 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.724792 17986 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.727592 17986 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:54.678535 18154 consensus_queue.cc:260] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.729789 18154 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:54.672192 17508 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:54.695726 17218 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.004s sys 0.023s
W20260807 08:01:54.734134 17218 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.004s sys 0.023s
I20260807 08:01:54.686738 17429 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:54.673246 17973 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.687191 18088 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:54.744757 14627 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s user 0.006s sys 0.027s
W20260807 08:01:54.745105 14627 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s user 0.006s sys 0.027s
W20260807 08:01:54.739657 14583 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.000s sys 0.032s
I20260807 08:01:54.736313 13709 raft_consensus.cc:2362] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7. Re-sending same reply.
W20260807 08:01:54.746337 14583 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.668s user 0.000s sys 0.033s
W20260807 08:01:54.753922 14580 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.009s sys 0.025s
W20260807 08:01:54.755774 14580 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.627s user 0.009s sys 0.025s
W20260807 08:01:54.756115 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 637004us
W20260807 08:01:54.750229 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 722599us
I20260807 08:01:54.761152 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:54.765267 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:54.752045 13540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.021s sys 0.043s
I20260807 08:01:54.764065 18113 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.768065 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:54.767657 18113 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:54.749496 17844 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.001s sys 0.061s
W20260807 08:01:54.768540 13540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.683s user 0.021s sys 0.043s
W20260807 08:01:54.749754 13537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.004s sys 0.029s
I20260807 08:01:54.770538 13701 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
W20260807 08:01:54.775574 13537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.691s user 0.005s sys 0.029s
W20260807 08:01:54.775897 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 1.30626e+06us
W20260807 08:01:54.777757 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 777288us
I20260807 08:01:54.783849 13540 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:54.773787 13661 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.025s sys 0.024s
W20260807 08:01:54.786206 13661 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.700s user 0.026s sys 0.024s
W20260807 08:01:54.786550 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 700926us
I20260807 08:01:54.788951 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:54.793215 13768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.103s user 0.008s sys 0.057s
W20260807 08:01:54.793929 13768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.127s user 0.008s sys 0.057s
W20260807 08:01:54.774405 17844 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s user 0.001s sys 0.061s
I20260807 08:01:54.781069 17585 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:54.769747 18152 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.067s sys 0.075s
I20260807 08:01:54.799292 18222 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:54.808411 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1609) took 1860 ms. Trace:
I20260807 08:01:54.825081 13807 rpcz_store.cc:276] 0807 08:01:52.948380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.948519 (+ 139us) service_pool.cc:227] Handling call
0807 08:01:52.948875 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:53.169137 (+220262us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.808146 (+1639009us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b540037c3d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:54.808350 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":498649472}
I20260807 08:01:54.801358 14619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 725) took 3365 ms. Trace:
I20260807 08:01:54.801420 18196 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:54.800688 13661 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:54.829648 13585 raft_consensus.cc:1226] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
W20260807 08:01:54.830821 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 289305us
I20260807 08:01:54.830983 13661 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:54.831400 13769 connection.cc:232] RPC call timeout handler was delayed by -1.2004971178248525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
I20260807 08:01:54.812809 17916 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.748762 17429 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.750813 18088 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:54.801330 13766 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.010s sys 0.014s
I20260807 08:01:54.782320 17997 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:54.825178 18152 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s user 0.068s sys 0.075s
I20260807 08:01:54.797621 17883 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:54.852511 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 57394.8us
I20260807 08:01:54.871714 13540 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:54.872781 13540 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:54.864768 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:54.837146 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1743971176911145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21850000000000003s for a response
I20260807 08:01:54.859830 13537 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:54.860318 13766 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.769s user 0.010s sys 0.015s
I20260807 08:01:54.874706 13807 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:54.861316 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:54.861935 18118 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.002s
I20260807 08:01:54.824157 18094 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:54.867573 13661 connection.cc:232] RPC call timeout handler was delayed by -1.3310873839072883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
I20260807 08:01:54.838753 14583 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:54.870883 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:54.834385 18196 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:54.869167 13538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.010s sys 0.027s
W20260807 08:01:54.883327 13538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.951s user 0.011s sys 0.027s
W20260807 08:01:54.882414 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1284971178974956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22360000000000002s for a response
I20260807 08:01:54.885661 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:54.886781 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:54.888428 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:54.855260 17585 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:54.896226 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:54.887564 14626 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 4.
I20260807 08:01:54.888837 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:54.889243 13711 raft_consensus.cc:2362] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7. Re-sending same reply.
I20260807 08:01:54.892706 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:54.893151 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:54.893448 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:54.898988 13805 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:54.894784 14625 raft_consensus.cc:1226] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:54.857838 18234 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:54.904374 13540 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:54.858601 17844 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.903986 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:54.917644 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:54.918154 13706 raft_consensus.cc:2362] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7. Re-sending same reply.
I20260807 08:01:54.906929 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:54.914448 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 604) took 1963 ms. Trace:
I20260807 08:01:54.856317 18219 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.906558 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:54.919402 14627 rpcz_store.cc:276] 0807 08:01:52.951032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.955721 (+ 4689us) service_pool.cc:227] Handling call
0807 08:01:52.956092 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:52.956346 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 08:01:54.005414 (+1049068us) log.cc:842] Serialized 14 byte log entry
0807 08:01:54.913733 (+908319us) log.cc:842] Serialized 14 byte log entry
0807 08:01:54.914027 (+ 294us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:54.914092 (+ 65us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:54.914157 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.914244 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:54.914372 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1217973,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":3321,"raft.run_cpu_time_us":589,"raft.run_wall_time_us":18909,"thread_start_us":1907249,"threads_started":3,"wal-append.queue_time_us":759551}
I20260807 08:01:54.926498 13579 raft_consensus.cc:2379] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:54.919811 13589 raft_consensus.cc:1261] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 08:01:54.919708 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:54.928030 13807 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:01:54.918967 13804 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:01:54.898379 13812 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 3.
I20260807 08:01:54.929435 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:54.929625 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:54.864694 18204 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.881043 17967 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.877719 17950 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:54.864795 18207 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.881953 18090 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:54.938889 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:54.950558 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 54176.4us
I20260807 08:01:54.919183 18210 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:54.937633 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:54.888142 18196 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:54.888451 18206 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:01:54.954912 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 116278us
I20260807 08:01:54.826658 14619 rpcz_store.cc:276] 0807 08:01:51.435878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:51.691084 (+255206us) service_pool.cc:227] Handling call
0807 08:01:51.716888 (+ 25804us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:52.438492 (+721604us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.780335 (+2341843us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b540043c6d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:54.799011 (+ 18676us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1524164352}
I20260807 08:01:54.876468 17997 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.956485 13708 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 08:01:54.957016 13805 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:54.989020 13661 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:54.990236 13805 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:54.885506 17872 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:01:54.895013 17958 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:54.964355 17524 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 08:01:54.946317 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 60240.2us
I20260807 08:01:54.959996 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:54.966477 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:54.916390 18113 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:54.903542 17508 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:54.915642 17986 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.924347 17218 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:54.862725 18222 consensus_queue.cc:1035] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.096s
W20260807 08:01:54.955011 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 50763.7us
I20260807 08:01:54.987725 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:55.018039 13592 raft_consensus.cc:2379] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 9: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:55.018396 13805 raft_consensus.cc:3032] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:54.886076 18094 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:54.859524 17916 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.957291 18224 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:55.001865 13589 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:54.965647 17967 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.026836 13589 raft_consensus.cc:1226] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:54.935916 18217 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.099s
I20260807 08:01:54.999536 14583 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:55.007687 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:54.965459 18009 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20260807 08:01:54.936880 17844 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:55.018050 13540 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:55.040416 13540 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:54.992669 13657 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:55.005322 17973 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.004433 17883 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:54.997570 17524 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:55.027690 13587 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:55.024695 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:54.950963 18219 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:55.018800 13805 raft_consensus.cc:729] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:01:55.062348 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 74024.7us
I20260807 08:01:54.960980 18204 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.982863 17950 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.008481 17958 raft_consensus.cc:686] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:55.046770 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 47357us
W20260807 08:01:55.033617 18240 log.cc:924] Time spent T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20260807 08:01:55.050019 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:55.055151 13804 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260807 08:01:55.051146 17883 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:55.105243 13804 pending_rounds.cc:76] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 6
I20260807 08:01:55.105592 13804 pending_rounds.cc:97] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260807 08:01:55.106020 13804 raft_consensus.cc:2866] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 08:01:55.060873 18238 log.cc:924] Time spent T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260807 08:01:55.022775 18118 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:55.074586 13700 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1585) took 1826 ms (client timeout 1807 ms). Trace:
I20260807 08:01:55.074661 13705 raft_consensus.cc:1226] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 8. Current term is 9. Ops: []
I20260807 08:01:55.043314 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:55.080504 13590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 618) took 2923 ms. Trace:
I20260807 08:01:55.071085 18218 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:55.126008 13590 rpcz_store.cc:276] 0807 08:01:52.156976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.158506 (+ 1530us) service_pool.cc:227] Handling call
0807 08:01:52.158782 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:52.961389 (+802607us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:55.080192 (+2118803us) spinlock_profiling.cc:243] Waited 795 ms on lock 0x7b540025f654. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:55.080432 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1828653952}
I20260807 08:01:55.074218 17617 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:55.062109 13805 consensus_queue.cc:260] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:54.983299 18207 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.053665 17872 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:01:55.058144 17524 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:55.138159 14620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:55594 (request call id 2) took 2810 ms. Trace:
I20260807 08:01:55.142867 14620 rpcz_store.cc:276] 0807 08:01:52.319080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.322962 (+ 3882us) service_pool.cc:227] Handling call
0807 08:01:55.106655 (+2783693us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5400698a54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf65bc3 00007f6b21eaf81a 00007f6b1b5a2d04 00007f6b1b5a2c92 00007f6b1b5a2bc1 00007f6b1b5a2b3c 00007f6b1b5a1df2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:55.124064 (+ 17409us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28741376}
I20260807 08:01:55.135353 18118 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.119139 13657 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:55.148752 13657 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:55.149333 17943 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:55.012751 17508 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.105336 13589 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:01:55.016032 17986 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.019711 17218 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.083213 18255 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:55.158594 18255 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:55.158185 17524 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.159296 18255 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.160367 17218 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:55.125103 13700 rpcz_store.cc:269] 0807 08:01:53.247651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:53.258069 (+ 10418us) service_pool.cc:227] Handling call
0807 08:01:55.074303 (+1816234us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54003aacd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:55.074499 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":424602496}
I20260807 08:01:55.161227 17524 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.131973 13805 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:55.091478 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 170222us
I20260807 08:01:55.118579 17883 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:55.104687 18143 log.cc:924] Time spent T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 2.045s user 0.005s sys 0.020s
I20260807 08:01:55.149971 17943 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:55.188400 17943 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:55.155323 18151 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:55.190241 18151 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:55.190867 18151 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:54.997679 18206 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.121243 18088 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.166349 17872 raft_consensus.cc:686] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [term 8 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:55.219828 13590 raft_consensus.cc:1261] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260807 08:01:55.178244 18245 log.cc:924] Time spent T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
W20260807 08:01:55.180030 18250 log.cc:924] Time spent T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
W20260807 08:01:55.186667 18228 log.cc:924] Time spent T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.238s user 0.000s sys 0.002s
I20260807 08:01:55.197794 18219 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:55.210026 17508 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.199465 18255 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:55.213114 17844 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:55.214787 17967 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.216325 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 459) took 2197 ms. Trace:
I20260807 08:01:55.045179 18224 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.218267 18204 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:55.222617 13805 raft_consensus.cc:1261] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260807 08:01:55.142653 17958 consensus_queue.cc:237] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:55.206722 17916 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:55.168309 17429 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.197650 17218 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:01:55.602101 13812 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.008s sys 0.008s
I20260807 08:01:55.668694 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:55.467147 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 468498us
I20260807 08:01:55.696346 13805 pending_rounds.cc:76] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 3
I20260807 08:01:55.696290 13807 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:55.700237 13805 pending_rounds.cc:97] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260807 08:01:55.666946 13537 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:55.631520 13540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.011s sys 0.006s
I20260807 08:01:55.663969 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:01:55.701654 13540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.575s user 0.013s sys 0.006s
I20260807 08:01:55.701757 13805 raft_consensus.cc:2866] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 08:01:55.702628 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 659341us
I20260807 08:01:55.661783 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
W20260807 08:01:55.632141 13535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.007s sys 0.006s
W20260807 08:01:55.703781 13535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.008s sys 0.006s
I20260807 08:01:55.432706 13710 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
W20260807 08:01:55.698776 13812 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s user 0.009s sys 0.009s
I20260807 08:01:55.244242 13702 rpcz_store.cc:276] 0807 08:01:53.018905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:53.019095 (+ 190us) service_pool.cc:227] Handling call
0807 08:01:53.062945 (+ 43850us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:53.096382 (+ 33437us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:53.096442 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:54.649214 (+1552772us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b2c000010a8. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1b94ee11 00007f6b1b95e442 00007f6b1cfa6c11 00007f6b1cf6b58a 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772
0807 08:01:54.667544 (+ 18330us) log.cc:842] Serialized 26 byte log entry
0807 08:01:54.784107 (+116563us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:54.902137 (+118030us) log.cc:842] Serialized 14 byte log entry
0807 08:01:54.950520 (+ 48383us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.950597 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:55.215929 (+265332us) raft_consensus.cc:1671] finished
0807 08:01:55.216064 (+ 135us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:55.216252 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43251,"prepare.queue_time_us":216454,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":186020,"raft.run_cpu_time_us":399,"raft.run_wall_time_us":24500,"spinlock_wait_cycles":66449280,"thread_start_us":213381,"threads_started":3,"wal-append.queue_time_us":192807}
I20260807 08:01:55.701944 14630 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:55.468264 13769 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.005s sys 0.008s
W20260807 08:01:55.706933 13769 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.010s sys 0.009s
I20260807 08:01:55.705041 13805 raft_consensus.cc:1226] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:55.202164 18009 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260807 08:01:55.713801 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 52127.5us
I20260807 08:01:55.717581 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:55.715732 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 591) took 2510 ms. Trace:
I20260807 08:01:55.239120 17499 raft_consensus.cc:3032] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
W20260807 08:01:55.354344 18211 consensus_meta.cc:287] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.657s user 0.014s sys 0.000s
W20260807 08:01:55.640295 18210 consensus_meta.cc:287] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.633s user 0.006s sys 0.009s
I20260807 08:01:55.719906 14629 rpcz_store.cc:276] 0807 08:01:53.205784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:53.205945 (+ 161us) service_pool.cc:227] Handling call
0807 08:01:53.206341 (+ 396us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:53.243136 (+ 36795us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:53.243189 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:53.243873 (+ 684us) log.cc:842] Serialized 52 byte log entry
0807 08:01:54.811668 (+1567795us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b1800416e60. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a025345 00007f6b22e5ca39 00007f6b1a0428bb 00007f6b1a040cbf 00007f6b1a042e22 00007f6b1cbec5bb 00007f6b1cbf2063 00007f6b1cbf2411 00007f6b1cf3f5cf
0807 08:01:54.872130 (+ 60462us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:54.872167 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:54.872229 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:55.307057 (+434828us) raft_consensus.cc:1671] finished
0807 08:01:55.307075 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:55.715665 (+408590us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13371,"raft.queue_time_us":28188,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":15060,"spinlock_wait_cycles":7601920,"thread_start_us":60287,"threads_started":1,"wal-append.queue_time_us":1739895}
I20260807 08:01:55.308269 18206 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.595376 18258 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:55.725242 13769 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.522s (SENT) [suppressed 18 similar messages]
W20260807 08:01:55.303687 18012 consensus_meta.cc:287] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 1.152s user 0.011s sys 0.005s
I20260807 08:01:55.664707 14620 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:55594
I20260807 08:01:55.301611 17872 consensus_queue.cc:237] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:55.242599 17997 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:55.721688 17499 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:55.720959 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:55.661197 18224 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.360355 18118 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:55.744612 13807 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:55.310833 17986 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.730486 18210 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:55.274947 18207 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:55.726775 18267 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:55.726262 17967 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:55.728201 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 61299.1us
I20260807 08:01:55.718251 18260 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:55.731433 18259 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:55.726434 17757 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:55.461736 18196 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:55.749434 13769 leader_election.cc:334] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 2.185s (SENT)
I20260807 08:01:55.660194 17883 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:55.758210 13587 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.011s sys 0.025s
I20260807 08:01:55.718701 18103 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
W20260807 08:01:55.781797 13587 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.542s user 0.011s sys 0.025s
W20260807 08:01:55.782276 13580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1805) took 2591 ms (client timeout 2192 ms). Trace:
I20260807 08:01:55.756982 18118 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:55.715953 17218 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.751063 14620 raft_consensus.cc:595] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:55.731997 18258 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.778506 13769 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:55.777216 17967 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:55.791468 13661 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:55.782745 13580 rpcz_store.cc:269] 0807 08:01:53.190771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.121803 (+931032us) service_pool.cc:227] Handling call
0807 08:01:55.782065 (+1660262us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b54002247d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:55.782188 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67479936}
I20260807 08:01:55.755939 17916 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:55.776379 18259 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:55.793219 13661 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:55.746017 18012 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.784986 17499 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.806671 17499 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:55.807008 18127 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:55.807525 18127 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:55.809350 17218 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.799413 17916 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.797224 18259 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:55.811734 17916 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.814427 18282 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:55.814944 18282 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:55.824847 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 1995) took 2516 ms. Trace:
I20260807 08:01:55.825325 13811 rpcz_store.cc:276] 0807 08:01:53.308011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:53.396937 (+ 88926us) service_pool.cc:227] Handling call
0807 08:01:53.404151 (+ 7214us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:54.575387 (+1171236us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:54.575427 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:54.576001 (+ 574us) log.cc:842] Serialized 26 byte log entry
0807 08:01:54.761884 (+185883us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:54.761958 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:55.794687 (+1032729us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54001fd1d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:55.794749 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:55.794813 (+ 64us) raft_consensus.cc:1671] finished
0807 08:01:55.794832 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:55.824766 (+ 29934us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":60187,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":83,"spinlock_wait_cycles":2692492160,"thread_start_us":156418,"threads_started":1,"wal-append.queue_time_us":170071,"wal-append.run_cpu_time_us":2596,"wal-append.run_wall_time_us":1002574}
I20260807 08:01:55.738018 18211 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:55.798780 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 83628.5us
I20260807 08:01:55.848045 13814 raft_consensus.cc:1226] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:55.815284 17967 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:55.796012 18118 raft_consensus.cc:686] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:55.851910 13816 raft_consensus.cc:1226] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
W20260807 08:01:55.852837 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 103430us
W20260807 08:01:55.815696 17524 consensus_queue.cc:1159] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9210dbc4cc9c45e5b26a463b705d362c" responder_term: 7 status { last_received { term: 4 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260807 08:01:55.854265 18279 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:01:55.815325 18282 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:55.859799 18196 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.809350 18127 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:55.873268 14629 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 08:01:55.860827 18005 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.782437 17757 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:55.843056 18163 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:55.885571 18163 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:55.886724 14620 raft_consensus.cc:1226] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:55.889844 18012 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.770226 18267 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.879325 14629 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 08:01:55.866681 17991 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:55.889768 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:55.841723 18206 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:55.899322 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:55.900449 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:55.901324 18253 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:55.778313 18260 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.892421 17991 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:55.912616 18260 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.916208 18013 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:55.873986 18103 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:55.873716 17883 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.863073 18277 consensus_queue.cc:1046] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.101s
I20260807 08:01:55.757476 13540 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:55.768639 13537 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:55.912616 17991 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:56.333948 17910 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.204461 13811 raft_consensus.cc:1261] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 08:01:56.337607 17844 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:56.269780 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 552067us
W20260807 08:01:56.180065 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 386887us
I20260807 08:01:56.196249 13700 raft_consensus.cc:1226] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:56.236387 13711 raft_consensus.cc:1226] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260807 08:01:56.337162 18016 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:56.234835 13805 raft_consensus.cc:1226] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: []
W20260807 08:01:56.181016 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 313556us
I20260807 08:01:56.336038 17915 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.166560 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1927) took 1171 ms. Trace:
I20260807 08:01:56.385936 17915 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.385599 13587 rpcz_store.cc:276] 0807 08:01:54.996582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:55.052020 (+ 55438us) service_pool.cc:227] Handling call
0807 08:01:55.167014 (+114994us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:55.167283 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 08:01:55.782389 (+615106us) log.cc:842] Serialized 14 byte log entry
0807 08:01:56.165583 (+383194us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:56.165724 (+ 141us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:56.165881 (+ 157us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.166024 (+ 143us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:56.166315 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":491892,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":173047,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":347,"thread_start_us":924964,"threads_started":2}
I20260807 08:01:56.176270 13592 raft_consensus.cc:1261] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 08:01:56.209147 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.223752 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.223404 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.189013 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.217782 13809 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 08:01:56.234376 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.247038 13804 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:56.332484 17924 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.400949 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:56.273145 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:56.004594 17950 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.164127 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260807 08:01:56.380579 17885 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260807 08:01:56.016647 18246 log.cc:924] Time spent T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 1.018s user 0.005s sys 0.010s
I20260807 08:01:56.381610 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.381765 17967 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:56.036839 18094 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.383315 18140 consensus_queue.cc:1046] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:56.384598 18222 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 08:01:56.334570 18103 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.291031 17681 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.389475 13587 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260807 08:01:55.972101 17943 consensus_meta.cc:287] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.783s user 0.013s sys 0.002s
I20260807 08:01:56.397728 17844 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:56.400084 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 642535us
I20260807 08:01:56.401378 13812 raft_consensus.cc:2396] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260807 08:01:56.130828 18282 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.403462 18268 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:55.930882 18151 consensus_meta.cc:287] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.736s user 0.011s sys 0.003s
I20260807 08:01:56.410406 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.411576 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 631) took 1356 ms. Trace:
I20260807 08:01:56.434340 18258 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.412392 13807 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
W20260807 08:01:56.413082 13584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1792) took 3282 ms (client timeout 1810 ms). Trace:
I20260807 08:01:56.414731 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1911) took 3448 ms. Trace:
I20260807 08:01:56.395704 13592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 634) took 1260 ms. Trace:
I20260807 08:01:56.286382 17654 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:56.422056 18267 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:55.972334 18118 consensus_queue.cc:237] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.422144 17967 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.422930 13590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 630) took 1374 ms. Trace:
I20260807 08:01:56.436230 13592 rpcz_store.cc:276] 0807 08:01:55.132877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:55.792908 (+660031us) service_pool.cc:227] Handling call
0807 08:01:56.175026 (+382118us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:56.391251 (+216225us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.393050 (+ 1799us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:01:56.424146 13580 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260807 08:01:56.424268 13706 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 08:01:56.390019 18016 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.331878 17757 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:56.430640 14619 raft_consensus.cc:2379] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 7: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:55.973901 18259 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.335610 17910 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.336733 17990 consensus_queue.cc:1035] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.097s
I20260807 08:01:56.091363 18211 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:56.436084 13585 rpcz_store.cc:276] 0807 08:01:52.966218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.967039 (+ 821us) service_pool.cc:227] Handling call
0807 08:01:52.973269 (+ 6230us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:54.848689 (+1875420us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.414193 (+1565504us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b5400077954. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:56.414503 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4265603968}
I20260807 08:01:56.433643 13589 rpcz_store.cc:276] 0807 08:01:55.055162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:55.151087 (+ 95925us) service_pool.cc:227] Handling call
0807 08:01:55.151501 (+ 414us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:55.307004 (+155503us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:55.307037 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:55.307075 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:55.307110 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.410930 (+1103820us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54001360d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:56.411025 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:56.411354 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":356994432}
I20260807 08:01:56.440158 13580 pending_rounds.cc:76] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad: Aborting all ops after (but not including) 7
W20260807 08:01:56.435772 13584 rpcz_store.cc:269] 0807 08:01:53.130412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:53.241411 (+110999us) service_pool.cc:227] Handling call
0807 08:01:56.412480 (+3171069us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54002ad854. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:56.412808 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3146419968}
I20260807 08:01:56.439453 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:55.928785 18163 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:56.440025 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.439442 13590 rpcz_store.cc:276] 0807 08:01:55.048177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:55.128566 (+ 80389us) service_pool.cc:227] Handling call
0807 08:01:55.128849 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:55.220376 (+ 91527us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.422440 (+1202064us) spinlock_profiling.cc:243] Waited 90.5 ms on lock 0x7b5400077454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:56.422775 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":208036736}
I20260807 08:01:56.401448 17924 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:56.453142 17885 raft_consensus.cc:2781] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:56.448239 13580 pending_rounds.cc:97] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260807 08:01:56.458858 13580 raft_consensus.cc:2866] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 08:01:56.456183 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 687472us
I20260807 08:01:56.453142 18094 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:56.456357 17885 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:56.470008 18293 consensus_queue.cc:1035] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:01:56.472049 17885 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:56.465971 18094 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:55.919435 18253 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.485754 18282 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.485771 18151 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:56.487490 14583 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:56.447018 18268 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.489006 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 658) took 1807 ms. Trace:
I20260807 08:01:56.489904 18151 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:56.492162 17844 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.497787 17967 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.490370 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 2007) took 1723 ms. Trace:
I20260807 08:01:56.490478 18113 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:56.498776 17943 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.498946 13817 rpcz_store.cc:276] 0807 08:01:54.766727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.766990 (+ 263us) service_pool.cc:227] Handling call
0807 08:01:54.767237 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:54.886947 (+119710us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:54.886989 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:54.887043 (+ 54us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:01:54.887096 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.489879 (+1602783us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54001b6554. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:56.489945 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:56.490207 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":274718592}
I20260807 08:01:56.490700 18268 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.484225 18094 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:56.489473 13814 rpcz_store.cc:276] 0807 08:01:54.681426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.681576 (+ 150us) service_pool.cc:227] Handling call
0807 08:01:54.681911 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:55.848656 (+1166745us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.488477 (+639821us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54001fc7d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:56.488795 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2681260928}
I20260807 08:01:56.521100 13592 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:56.523388 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.472008 17681 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.525799 14583 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:56.525883 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:56.526252 13702 raft_consensus.cc:1261] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 08:01:56.478502 17915 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.531592 18103 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:56.532223 17681 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.532325 18103 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.535936 18113 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:56.508950 18016 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.537391 13661 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:56.549005 13588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 627) took 3587 ms. Trace:
I20260807 08:01:56.479653 17654 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.553416 13588 rpcz_store.cc:276] 0807 08:01:52.955299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:52.955454 (+ 155us) service_pool.cc:227] Handling call
0807 08:01:52.955786 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:54.074207 (+1118421us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.538761 (+2464554us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400076554. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:56.542315 (+ 3554us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2556865920}
W20260807 08:01:56.555162 13701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1821) took 1877 ms (client timeout 1548 ms). Trace:
I20260807 08:01:56.562661 14583 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:56.563241 18230 consensus_queue.cc:1046] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:56.564692 18230 raft_consensus.cc:3032] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:56.565892 18113 raft_consensus.cc:2781] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:56.566555 18113 raft_consensus.cc:686] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:56.566924 17910 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:01:56.562487 13701 rpcz_store.cc:269] 0807 08:01:54.677299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.677439 (+ 140us) service_pool.cc:227] Handling call
0807 08:01:56.554707 (+1877268us) spinlock_profiling.cc:243] Waited 92 ms on lock 0x7b54002ff154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:56.554876 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":211625856}
W20260807 08:01:56.568966 13709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1822) took 1891 ms (client timeout 1578 ms). Trace:
I20260807 08:01:56.569481 18253 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:56.569831 13537 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:56.571981 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.572732 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 635) took 1439 ms. Trace:
I20260807 08:01:56.572868 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.573786 13706 raft_consensus.cc:2379] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:56.574185 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:56.569391 13709 rpcz_store.cc:269] 0807 08:01:54.677918 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.678030 (+ 112us) service_pool.cc:227] Handling call
0807 08:01:56.568554 (+1890524us) spinlock_profiling.cc:243] Waited 50.1 ms on lock 0x7b5400408154. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:56.568744 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115316992}
I20260807 08:01:56.573118 13580 rpcz_store.cc:276] 0807 08:01:55.133543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:56.165743 (+1032200us) service_pool.cc:227] Handling call
0807 08:01:56.423337 (+257594us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:56.459857 (+ 36520us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.572513 (+112656us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54002ad850. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:56.572654 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":572910336}
I20260807 08:01:56.577598 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1920) took 1616 ms. Trace:
I20260807 08:01:56.576575 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:56.578650 13701 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 8: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:56.567950 18113 consensus_queue.cc:237] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.578033 13579 rpcz_store.cc:276] 0807 08:01:54.926006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.927415 (+ 1409us) service_pool.cc:227] Handling call
0807 08:01:54.940290 (+ 12875us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:55.081387 (+141097us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:55.081442 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:55.081497 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:55.081564 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.542154 (+1460590us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54002add54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:56.542229 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:56.542445 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":323921152}
I20260807 08:01:56.565038 14583 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:56.579850 18196 raft_consensus.cc:3032] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:56.579676 18103 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:56.580749 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 68933.5us
I20260807 08:01:56.565213 18230 raft_consensus.cc:729] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:56.582553 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 116190us
I20260807 08:01:56.583263 18291 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 08:01:56.591190 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:56.592549 18268 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:56.557628 17924 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:01:56.598179 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 158720us
I20260807 08:01:56.607323 18285 consensus_queue.cc:1046] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 08:01:56.607611 18168 consensus_queue.cc:1035] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:56.608103 13657 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:56.608807 14619 raft_consensus.cc:2396] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260807 08:01:56.608459 18127 consensus_queue.cc:1046] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:56.609985 13661 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:56.559738 17943 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.602327 18196 raft_consensus.cc:729] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:56.621857 13537 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:56.624939 18259 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:56.626368 18142 raft_consensus.cc:2726] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:56.626678 18222 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:56.626448 18259 raft_consensus.cc:3032] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:56.635885 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.635835 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:56.642210 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 44602.1us
I20260807 08:01:56.642959 13814 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:56.643983 18291 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:56.627266 18222 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:01:56.609839 18230 consensus_queue.cc:260] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.628527 18259 raft_consensus.cc:729] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:56.630633 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.649130 17910 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:01:56.651015 13661 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:56.650794 18196 consensus_queue.cc:260] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:56.647389 17844 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:56.678544 17844 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:56.659422 18103 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:56.665797 17924 consensus_queue.cc:1046] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:56.669174 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:01:56.636431 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.631851 18268 raft_consensus.cc:686] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:56.670127 17654 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.612301 17915 consensus_queue.cc:1035] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260807 08:01:56.644492 18291 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:56.676312 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:56.685238 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 120430us
I20260807 08:01:56.680836 17844 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:56.701843 18291 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:56.712839 13809 raft_consensus.cc:2396] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260807 08:01:56.676115 18222 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:56.713685 18103 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.724961 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.696241 17943 consensus_queue.cc:1046] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260807 08:01:56.728240 13817 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:56.730046 18118 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.730177 18103 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.731961 18118 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.732457 14619 raft_consensus.cc:1226] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260807 08:01:56.733794 18103 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:56.730906 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:56.712327 17910 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:56.734097 18186 consensus_queue.cc:1046] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260807 08:01:56.744218 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1814) took 2662 ms. Trace:
I20260807 08:01:56.744648 13705 rpcz_store.cc:276] 0807 08:01:54.082059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.082268 (+ 209us) service_pool.cc:227] Handling call
0807 08:01:55.074434 (+992166us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:55.129615 (+ 55181us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.743847 (+1614232us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54003aacd0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:56.744031 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2280929536}
W20260807 08:01:56.749281 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 98476.1us
I20260807 08:01:56.734126 18118 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:56.739253 18196 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:56.751796 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.721982 17885 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:56.754595 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1935) took 1001 ms. Trace:
I20260807 08:01:56.740571 17943 consensus_queue.cc:1225] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Peer 77da774f2bb54cf28302d53d0db224ad log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260807 08:01:56.762307 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:56.762416 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:56.763033 18259 consensus_queue.cc:260] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.767414 18222 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:56.775996 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.777953 13711 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:56.778177 18259 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:56.746912 18087 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260807 08:01:56.781667 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:56.747318 18253 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.037s
I20260807 08:01:56.789309 18268 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.789880 13701 raft_consensus.cc:1261] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260807 08:01:56.794091 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260807 08:01:56.794216 18013 consensus_meta.cc:287] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.877s user 0.010s sys 0.018s
I20260807 08:01:56.796150 13711 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:01:56.799196 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:56.800576 17885 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:56.800736 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:56.802690 17429 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.811968 13809 raft_consensus.cc:2379] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:56.811530 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:56.800912 13710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 471) took 1877 ms (client timeout 2172 ms). Trace:
I20260807 08:01:56.769312 13585 rpcz_store.cc:276] 0807 08:01:55.753287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:56.457148 (+703861us) service_pool.cc:227] Handling call
0807 08:01:56.469772 (+ 12624us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:56.470286 (+ 514us) raft_consensus.cc:1520] Early marking committed up to index 5
0807 08:01:56.635514 (+165228us) log.cc:842] Serialized 14 byte log entry
0807 08:01:56.754164 (+118650us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:01:56.754229 (+ 65us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:01:56.754293 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:56.754375 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:56.754525 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":279825,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":50,"thread_start_us":271224,"threads_started":2}
I20260807 08:01:56.820612 13811 raft_consensus.cc:2454] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:56.820837 13814 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:56.800797 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:56.820492 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:56.822820 13817 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
W20260807 08:01:56.817282 13710 rpcz_store.cc:269] 0807 08:01:54.890997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.896015 (+ 5018us) service_pool.cc:227] Handling call
0807 08:01:56.747590 (+1851575us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b54000cf654. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:56.767979 (+ 20389us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":609524096}
I20260807 08:01:56.826387 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:56.826786 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.827122 14620 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:56.832182 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:56.808634 17429 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.834867 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.835685 13706 raft_consensus.cc:2379] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:56.839397 13808 raft_consensus.cc:2396] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260807 08:01:56.840204 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:56.843864 18094 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.846107 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:56.847203 18230 raft_consensus.cc:3037] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:56.849404 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.842304 18016 consensus_queue.cc:1046] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:56.834110 17844 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.864248 13592 raft_consensus.cc:2396] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:01:56.865208 13809 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:56.866009 13579 raft_consensus.cc:2379] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 7: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
W20260807 08:01:56.867363 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 155444us
W20260807 08:01:56.869603 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 156980us
I20260807 08:01:56.873050 13804 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:56.850426 17924 raft_consensus.cc:3032] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:01:56.872795 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.883927 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:56.862278 18222 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:56.890609 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:56.890918 18222 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:56.891418 13588 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:01:56.892244 18094 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.877156 17924 raft_consensus.cc:729] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:56.893096 18222 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:56.893589 17924 consensus_queue.cc:260] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:56.894984 17924 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:56.896273 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56576 (request call id 1012) took 3895 ms. Trace:
I20260807 08:01:56.900576 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:56.901872 18254 consensus_queue.cc:1046] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:56.902323 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.902944 18254 raft_consensus.cc:3032] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:01:56.903328 18254 raft_consensus.cc:729] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:56.903376 18201 consensus_queue.cc:1046] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 08:01:56.903786 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.904347 18254 consensus_queue.cc:260] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:56.905125 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.905876 13809 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:01:56.906613 17991 consensus_queue.cc:1046] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260807 08:01:56.901783 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.908303 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:56.908972 13588 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
W20260807 08:01:56.909670 13808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 1994) took 3611 ms (client timeout 2441 ms). Trace:
I20260807 08:01:56.910036 18094 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:56.911880 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.913331 13701 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:56.913720 18222 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:56.914662 18206 raft_consensus.cc:3032] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:56.915091 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:56.915107 18206 raft_consensus.cc:729] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:56.876044 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:56.914400 18222 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:56.917599 18206 consensus_queue.cc:260] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.918373 14619 raft_consensus.cc:2379] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:01:56.918885 18206 raft_consensus.cc:3037] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:56.921240 18277 consensus_queue.cc:1035] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:01:56.898608 13591 rpcz_store.cc:276] 0807 08:01:53.000564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:53.001405 (+ 841us) service_pool.cc:227] Handling call
0807 08:01:56.893982 (+3892577us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54001e4e54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0af0 00007f6b1cf73f2a 00007f6b21eb0c6e 00007f6b1b5a7807 00007f6b1b5a7792 00007f6b1b5a76c1 00007f6b1b5a763c 00007f6b1b5a68f2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:56.896028 (+ 2046us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4232565632}
I20260807 08:01:56.897284 14622 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260807 08:01:56.909426 13769 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:56.898180 18285 raft_consensus.cc:979] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
W20260807 08:01:56.910072 13808 rpcz_store.cc:269] 0807 08:01:53.298343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:53.298498 (+ 155us) service_pool.cc:227] Handling call
0807 08:01:56.909377 (+3610879us) spinlock_profiling.cc:243] Waited 3.54 s on lock 0x7b54001fbdd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:56.909562 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8139910016}
W20260807 08:01:56.948276 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 80727.1us
I20260807 08:01:56.951197 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:56.954398 18303 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.955008 18303 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:56.896310 14620 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:56.956696 14630 raft_consensus.cc:1226] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260807 08:01:56.963450 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:56.944698 17844 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.963536 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:56.965893 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:56.975363 13657 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:56.963533 18267 raft_consensus.cc:2741] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260807 08:01:56.990636 18161 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:56.990763 17429 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:56.997479 13808 raft_consensus.cc:1261] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 08:01:56.970459 18254 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:56.993611 13657 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:56.994912 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:56.999523 13580 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:56.995338 13701 raft_consensus.cc:3032] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:56.994966 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:57.001304 13701 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:57.009596 18308 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:57.003640 18076 raft_consensus.cc:2781] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:01:57.000042 18222 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:57.015050 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 105740us
I20260807 08:01:57.014755 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.013840 18308 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.025969 18303 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:57.027159 18161 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.027798 18308 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:57.024921 13701 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.029914 18161 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:57.030529 13769 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:57.006171 18140 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:57.036780 13812 raft_consensus.cc:1261] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260807 08:01:57.039093 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 122731us
I20260807 08:01:57.041769 18076 raft_consensus.cc:686] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [term 8 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:57.042373 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:57.043334 13808 raft_consensus.cc:2396] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260807 08:01:57.031576 18255 raft_consensus.cc:1067] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:01:57.039441 18267 raft_consensus.cc:2741] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260807 08:01:57.071115 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.069793 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:57.056295 18222 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:57.075266 17429 consensus_queue.cc:1046] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:57.075340 18222 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.076457 17429 raft_consensus.cc:3032] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 08:01:57.076797 17429 raft_consensus.cc:729] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:57.077854 17429 consensus_queue.cc:260] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.078094 18076 consensus_queue.cc:237] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.079293 17429 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:57.051788 18140 raft_consensus.cc:686] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:57.086261 18140 consensus_queue.cc:237] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.087520 13537 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:57.099923 13657 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:57.102459 18303 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:57.104858 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.104422 18303 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:57.113935 18303 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:57.117897 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.118842 13592 raft_consensus.cc:2454] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:57.122556 17844 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:57.123025 18196 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:57.123164 18315 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:57.107425 18319 consensus_queue.cc:1035] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20260807 08:01:57.123864 18315 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.102998 18222 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:57.125793 13580 raft_consensus.cc:2362] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7. Re-sending same reply.
I20260807 08:01:57.126132 18315 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:57.130160 18230 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 08:01:57.131230 17681 consensus_queue.cc:1046] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:01:57.132916 18230 raft_consensus.cc:3032] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:57.133860 18230 raft_consensus.cc:729] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:57.137055 18230 consensus_queue.cc:260] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.141274 18230 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:57.104487 18308 consensus_queue.cc:1035] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260807 08:01:57.070966 18291 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.147720 18291 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:57.147779 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.148722 13804 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:01:57.135842 14630 raft_consensus.cc:1261] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 08:01:57.138209 14622 raft_consensus.cc:1226] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260807 08:01:57.138784 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.150641 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:57.133869 18190 raft_consensus.cc:3032] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:57.135478 13816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 2004) took 2372 ms. Trace:
I20260807 08:01:57.149894 13537 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:57.157424 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.150861 14583 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:57.153798 13816 rpcz_store.cc:276] 0807 08:01:54.762954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.763057 (+ 103us) service_pool.cc:227] Handling call
0807 08:01:54.763321 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:55.862785 (+1099464us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:57.135088 (+1272303us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400125f54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:57.135310 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2503046016}
I20260807 08:01:57.170580 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:57.170190 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:57.173419 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.175031 13812 raft_consensus.cc:2379] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:57.177269 13588 raft_consensus.cc:2379] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:57.177784 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.179697 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:57.180295 13591 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:57.154690 18190 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:57.182598 14583 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:57.181471 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.161486 18268 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:01:57.184662 13580 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:57.185886 13579 raft_consensus.cc:2379] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:57.186450 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.187132 13590 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:57.188087 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:57.188695 14630 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:57.188895 18255 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:57.189309 18255 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:57.189723 18255 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:57.194398 18310 log.cc:924] Time spent T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.281s user 0.003s sys 0.000s
I20260807 08:01:57.198648 14630 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
W20260807 08:01:57.203650 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 109448us
I20260807 08:01:57.177347 18319 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:57.161180 18285 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260807 08:01:57.161021 18113 consensus_queue.cc:1046] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20260807 08:01:57.213361 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.211552 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:57.198812 18268 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:57.217831 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:57.218437 13702 raft_consensus.cc:3037] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:57.215453 18326 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:57.220443 18326 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:57.220764 18326 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:57.221837 13705 raft_consensus.cc:2354] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
I20260807 08:01:57.226464 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.227602 13706 raft_consensus.cc:2396] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260807 08:01:57.229264 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1927168781403452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
I20260807 08:01:57.240942 13702 raft_consensus.cc:2454] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:57.241925 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.237329 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.255437 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:57.256837 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1900168783031404s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
I20260807 08:01:57.265758 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:57.269362 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 86877.6us
I20260807 08:01:57.275739 14583 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:57.276166 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.238396 18268 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:57.279695 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.284067 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.294410 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.294615 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:57.294771 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.295181 13705 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:01:57.291908 13592 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:57.296900 13579 raft_consensus.cc:2354] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 8.
I20260807 08:01:57.295293 13711 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:57.301337 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:57.293807 13702 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:01:57.307468 13587 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:57.308139 13585 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 8. Current term is 9. Ops: []
I20260807 08:01:57.309917 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.283414 18277 raft_consensus.cc:2741] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260807 08:01:57.317817 13701 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:01:57.314565 14583 leader_election.cc:334] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033): Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 2.172s (SENT)
I20260807 08:01:57.281291 18087 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:57.319864 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.323414 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.323520 14583 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:57.323952 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:57.327051 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.328487 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:57.329435 13587 raft_consensus.cc:1226] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260807 08:01:57.287901 17943 consensus_queue.cc:1046] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260807 08:01:57.331684 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.336604 14630 raft_consensus.cc:1261] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 08:01:57.339162 14622 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:57.339957 13807 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 08:01:57.341126 13710 raft_consensus.cc:1226] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260807 08:01:57.341650 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:57.342561 14627 raft_consensus.cc:2396] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260807 08:01:57.345937 18087 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.331401 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.348434 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.416385 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.347734 13579 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:57.358489 13814 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:57.384303 13591 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:01:57.347805 13703 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:57.420429 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:57.410492 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:57.421734 13589 raft_consensus.cc:3037] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:01:57.427515 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 118692us
I20260807 08:01:57.430015 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.359227 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.353156 13585 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260807 08:01:57.434370 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:57.416514 18333 log.cc:924] Time spent T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260807 08:01:57.441483 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.338868 18201 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:57.442840 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.431550 13585 pending_rounds.cc:76] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Aborting all ops after (but not including) 7
I20260807 08:01:57.443069 14627 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:57.444140 13585 pending_rounds.cc:97] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260807 08:01:57.444420 13585 raft_consensus.cc:2866] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:57.445086 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.444666 13584 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:01:57.427119 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.467259 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.468164 13711 raft_consensus.cc:1226] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:57.468479 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.469130 13701 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 08:01:57.466639 13703 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20260807 08:01:57.486692 13808 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:57.470939 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.484056 13702 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:57.469978 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.488173 14620 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:01:57.471995 13580 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:57.455885 14630 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:01:57.477851 14622 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:57.456959 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.490481 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.491353 13701 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:57.471019 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.450445 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:57.470592 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.488751 14627 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
W20260807 08:01:57.466382 18206 consensus_meta.cc:287] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.547s user 0.010s sys 0.000s
I20260807 08:01:57.506583 14618 raft_consensus.cc:2396] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:01:57.492417 13700 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:01:57.442786 17943 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:57.453727 18327 log.cc:924] Time spent T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20260807 08:01:57.402281 17885 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260807 08:01:57.350085 18303 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.447678 13814 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:59332
I20260807 08:01:57.461920 18319 consensus_queue.cc:1046] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.182s
I20260807 08:01:57.468277 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.695401 18186 raft_consensus.cc:3032] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:01:57.695887 18186 raft_consensus.cc:729] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:57.471704 13708 raft_consensus.cc:1226] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:57.537839 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:57.458191 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.462649 18337 consensus_queue.cc:1046] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.237s
I20260807 08:01:57.455721 18255 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.699947 18345 raft_consensus.cc:3032] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:01:57.700366 18345 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:57.700377 14629 raft_consensus.cc:1226] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:57.701427 18345 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.702435 18255 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:57.702620 18345 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:57.476039 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:57.492605 18168 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260807 08:01:57.469449 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.705188 13587 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
W20260807 08:01:57.447609 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 166009us
W20260807 08:01:57.707929 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1603168782312423s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
W20260807 08:01:57.709849 13769 connection.cc:232] RPC call timeout handler was delayed by -1.1207168782129884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
I20260807 08:01:57.581892 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.710122 18156 raft_consensus.cc:3032] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:01:57.712090 18156 raft_consensus.cc:729] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:57.481539 18190 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.604151 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.642328 13805 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260807 08:01:57.729074 13805 pending_rounds.cc:76] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 4
I20260807 08:01:57.729379 13805 pending_rounds.cc:97] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260807 08:01:57.729701 13805 raft_consensus.cc:2866] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:57.720204 13591 raft_consensus.cc:1261] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260807 08:01:57.722314 13814 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:01:57.730379 13814 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:57.662577 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.661932 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.731482 14626 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:01:57.696858 18186 consensus_queue.cc:260] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.732389 14625 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:57.732836 13585 raft_consensus.cc:2454] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 2.
I20260807 08:01:57.692308 18344 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.241s
I20260807 08:01:57.733816 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:57.734735 13702 raft_consensus.cc:2396] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260807 08:01:57.735584 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.736631 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:57.737110 13585 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:57.737843 18319 raft_consensus.cc:3032] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 08:01:57.738312 18319 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:57.739230 18319 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.740391 13589 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:01:57.740572 18319 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:57.741250 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.742144 13804 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:57.743161 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.732633 18254 consensus_queue.cc:1046] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:01:57.743162 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:57.641583 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.743726 18294 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 08:01:57.745599 14630 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:57.641995 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:57.450101 18087 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:57.673324 18339 log.cc:924] Time spent T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.338s user 0.001s sys 0.000s
I20260807 08:01:57.750243 13589 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:01:57.585912 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:57.756120 13808 raft_consensus.cc:1226] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:57.488865 18230 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:57.761873 13588 raft_consensus.cc:2379] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 6: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:57.765259 13804 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260807 08:01:57.765766 13804 pending_rounds.cc:76] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 6
I20260807 08:01:57.766076 13804 pending_rounds.cc:97] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260807 08:01:57.766366 13804 raft_consensus.cc:2866] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:57.769872 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.712989 18156 consensus_queue.cc:260] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.778268 13584 raft_consensus.cc:1226] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:57.780323 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:57.761734 18343 log.cc:924] Time spent T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.354s user 0.001s sys 0.002s
I20260807 08:01:57.786283 13808 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:57.468165 17218 consensus_queue.cc:1046] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.320s
I20260807 08:01:57.465437 18326 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.798528 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:57.813486 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.821357 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:57.767617 18206 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260807 08:01:57.834578 17585 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.033s
I20260807 08:01:57.800493 18190 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:57.878857 13809 raft_consensus.cc:1261] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260807 08:01:57.879906 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:57.754384 17943 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:57.885337 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 66385.4us
W20260807 08:01:57.889453 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 62139.2us
I20260807 08:01:57.801659 18337 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260807 08:01:57.785424 18308 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260807 08:01:57.815106 18291 consensus_queue.cc:1046] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.075s
W20260807 08:01:57.879243 18343 log.cc:924] Time spent T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260807 08:01:57.898708 13589 raft_consensus.cc:2379] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 9: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:57.780195 18230 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.917596 13814 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:57.906912 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 161898us
I20260807 08:01:57.923928 18090 raft_consensus.cc:3032] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:57.924351 18090 raft_consensus.cc:729] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:57.925235 18090 consensus_queue.cc:260] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:57.926501 18090 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:57.928982 13588 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260807 08:01:57.932745 18268 consensus_meta.cc:287] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.620s user 0.009s sys 0.000s
I20260807 08:01:57.922546 13814 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:57.942456 14619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 607) took 1020 ms. Trace:
I20260807 08:01:57.942871 14619 rpcz_store.cc:276] 0807 08:01:56.922746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:56.932205 (+ 9459us) service_pool.cc:227] Handling call
0807 08:01:56.944786 (+ 12581us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:56.954892 (+ 10106us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:56.954922 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:56.955580 (+ 658us) log.cc:842] Serialized 52 byte log entry
0807 08:01:57.175887 (+220307us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:57.179410 (+ 3523us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:57.202830 (+ 23420us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:57.919531 (+716701us) raft_consensus.cc:1671] finished
0807 08:01:57.920301 (+ 770us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:57.942370 (+ 22069us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6234,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":10894,"thread_start_us":205100,"threads_started":1,"wal-append.queue_time_us":338928}
I20260807 08:01:57.890136 18087 consensus_queue.cc:1035] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260807 08:01:57.933216 18268 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:57.973546 18308 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:58.003410 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:58.007066 13580 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 9: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:01:57.947602 18320 consensus_queue.cc:1035] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260807 08:01:58.012486 18303 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.034018 13540 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:01:58.052241 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 139222us
W20260807 08:01:58.051052 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 235570us
I20260807 08:01:58.007764 18308 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:58.055722 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:58.056962 13592 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:58.034160 18367 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:58.054527 13540 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:58.063782 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:58.065097 13811 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:58.068859 13769 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:58.052217 18087 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:58.097222 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:58.098893 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 44437.8us
I20260807 08:01:58.064796 18326 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.104486 13540 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:58.094144 18090 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:58.108467 17943 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:58.098352 18303 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:58.109465 18308 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.109216 18303 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.111296 18374 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:58.111793 18374 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:58.112150 18374 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:58.070035 18294 consensus_queue.cc:1035] T 8a8da02e8f51481cbcca504f8cff6f80 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.044s
I20260807 08:01:58.111326 18303 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:58.115073 13589 raft_consensus.cc:1226] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 7. Current term is 8. Ops: []
W20260807 08:01:58.118065 18361 log.cc:924] Time spent T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260807 08:01:58.119692 18376 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:58.120774 18230 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:58.124543 13540 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:58.125370 18376 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:58.128763 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:58.128993 13585 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
W20260807 08:01:58.146096 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 63619.3us
I20260807 08:01:58.156931 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:58.157817 13805 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:58.159211 13537 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:58.159842 14619 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260807 08:01:58.160687 18090 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:58.161108 18090 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:01:58.162801 13769 leader_election.cc:334] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.578s (SENT)
W20260807 08:01:58.163777 13769 leader_election.cc:334] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.581s (SENT)
I20260807 08:01:58.164136 13769 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:58.165041 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:58.165637 14620 raft_consensus.cc:3032] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Stepping down as leader of term 2
I20260807 08:01:58.165925 14620 raft_consensus.cc:729] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:58.170182 13592 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:01:58.173877 13540 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:58.174113 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:58.172241 14583 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:58.147878 18268 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.161424 18090 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:58.173878 14620 consensus_queue.cc:260] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.155460 18308 consensus_queue.cc:1035] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.071s
I20260807 08:01:58.151229 18374 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:58.306027 18379 log.cc:924] Time spent T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260807 08:01:58.307165 13702 raft_consensus.cc:1261] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260807 08:01:58.305346 18371 log.cc:924] Time spent T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20260807 08:01:58.154552 18367 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:58.333864 13579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1944) took 1006 ms. Trace:
I20260807 08:01:58.309195 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.181064 18253 raft_consensus.cc:3032] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Stepping down as leader of term 7
W20260807 08:01:58.221509 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 68582.5us
I20260807 08:01:58.335438 13805 raft_consensus.cc:1226] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:58.198401 18319 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:01:58.186218 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:58.341835 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:58.309149 13708 raft_consensus.cc:1261] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 08:01:58.226073 18268 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:58.334446 13579 rpcz_store.cc:276] 0807 08:01:57.327535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:57.704827 (+377292us) service_pool.cc:227] Handling call
0807 08:01:57.705195 (+ 368us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:57.748195 (+ 43000us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:57.748235 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:57.780674 (+ 32439us) log.cc:842] Serialized 26 byte log entry
0807 08:01:58.030177 (+249503us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:01:58.030397 (+ 220us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:58.030483 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:58.333628 (+303145us) raft_consensus.cc:1671] finished
0807 08:01:58.333651 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:58.333790 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":42605,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8194,"thread_start_us":199974,"threads_started":1,"wal-append.queue_time_us":251019}
I20260807 08:01:58.335218 18253 raft_consensus.cc:729] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:58.334954 13701 raft_consensus.cc:2379] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 8: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:58.193791 18254 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:58.372048 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:58.373188 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:58.374378 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.375795 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:01:58.308619 18372 log.cc:924] Time spent T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260807 08:01:58.373946 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.378930 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:58.379140 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:58.378901 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.380679 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.377240 18268 raft_consensus.cc:686] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:58.381834 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 218074us
I20260807 08:01:58.382530 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:58.382993 18268 consensus_queue.cc:237] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.384672 18374 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:58.385790 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:58.387214 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:58.375476 14583 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:58.389423 14625 raft_consensus.cc:2379] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 8: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
W20260807 08:01:58.382649 13769 leader_election.cc:334] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.614s (SENT)
I20260807 08:01:58.390475 13769 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:58.382071 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:58.337762 18367 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:58.399829 13702 raft_consensus.cc:2379] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 8: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:01:58.248831 18113 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:58.396763 14626 raft_consensus.cc:2396] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260807 08:01:58.398584 14622 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:58.337471 18087 consensus_queue.cc:1046] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260807 08:01:58.404070 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.406836 13585 raft_consensus.cc:1226] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:01:58.365865 18319 raft_consensus.cc:686] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 8 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:58.409924 13706 raft_consensus.cc:1261] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260807 08:01:58.414348 13579 raft_consensus.cc:1226] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 9. Current term is 10. Ops: []
I20260807 08:01:58.410288 13706 pending_rounds.cc:76] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: Aborting all ops after (but not including) 3
I20260807 08:01:58.417793 13706 pending_rounds.cc:97] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260807 08:01:58.418128 13706 raft_consensus.cc:2866] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:58.418577 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.416783 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:58.430948 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 621) took 1100 ms. Trace:
I20260807 08:01:58.406641 18259 raft_consensus.cc:2741] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260807 08:01:58.431497 14629 rpcz_store.cc:276] 0807 08:01:57.318604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:57.318732 (+ 128us) service_pool.cc:227] Handling call
0807 08:01:57.326781 (+ 8049us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:57.701012 (+374231us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:58.418619 (+717607us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b5400073354. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:01:58.418793 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":858580864}
I20260807 08:01:58.503785 13579 raft_consensus.cc:1261] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260807 08:01:58.410096 18291 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:58.410247 18140 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:58.500142 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 90475.9us
I20260807 08:01:58.506062 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.501332 14622 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:58.402829 18367 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:58.412675 18230 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:58.388252 18344 consensus_queue.cc:1046] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260807 08:01:58.499186 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:01:58.517977 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 135372us
I20260807 08:01:58.517905 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:58.414660 18374 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:58.520862 13700 raft_consensus.cc:1261] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260807 08:01:58.515954 18357 log.cc:924] Time spent T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.588s user 0.002s sys 0.000s
I20260807 08:01:58.522042 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:58.418687 18076 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:58.645730 14625 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260807 08:01:58.413635 18113 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.657936 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:58.658562 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.663010 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:58.663849 13708 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:58.506011 18380 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:58.667207 13769 leader_election.cc:334] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.658s (SENT)
I20260807 08:01:58.671759 13769 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:58.664912 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 259528us
I20260807 08:01:58.646572 18087 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:58.677145 18391 log.cc:924] Time spent T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.153s user 0.001s sys 0.000s
I20260807 08:01:58.512519 18291 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:58.505573 18253 consensus_queue.cc:260] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.708112 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1652) took 1580 ms. Trace:
I20260807 08:01:58.654011 18230 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:58.684929 17915 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:58.714541 13817 rpcz_store.cc:276] 0807 08:01:57.127590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:57.339242 (+211652us) service_pool.cc:227] Handling call
0807 08:01:57.403061 (+ 63819us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:01:57.437321 (+ 34260us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:01:57.437353 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:01:57.478053 (+ 40700us) log.cc:842] Serialized 527 byte log entry
0807 08:01:57.764411 (+286358us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:01:57.764445 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:57.764511 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:58.515153 (+750642us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:58.701827 (+186674us) raft_consensus.cc:1671] finished
0807 08:01:58.701849 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:58.708044 (+ 6195us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17254,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":4058,"spinlock_wait_cycles":40060160,"thread_start_us":286037,"threads_started":1,"wal-append.queue_time_us":421114}
I20260807 08:01:58.699437 18344 raft_consensus.cc:3032] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:58.719810 13702 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
W20260807 08:01:58.678313 18303 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.054s sys 0.108s
I20260807 08:01:58.523183 18140 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.676539 18076 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.702771 18259 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:58.725008 17915 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:58.739033 13710 raft_consensus.cc:1261] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260807 08:01:58.740295 13769 leader_election.cc:398] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Vote denied by peer 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033) with higher term. Message: Invalid argument: T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 5. Current term is 6.
W20260807 08:01:58.739303 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 321484us
I20260807 08:01:58.678138 18380 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:58.734890 18303 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.570s user 0.054s sys 0.108s
I20260807 08:01:58.702445 18087 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:58.734239 18253 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:58.730073 18344 raft_consensus.cc:729] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:58.695065 18319 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.734208 18254 consensus_queue.cc:1035] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260807 08:01:58.766904 13661 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:58.768857 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 64398.7us
I20260807 08:01:58.769758 13661 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:58.771073 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 48838.6us
I20260807 08:01:58.768178 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:58.772047 13769 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:58.743212 18259 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.773736 18398 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:58.773969 18399 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:58.784842 13769 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:58.785009 18140 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:58.787307 18376 raft_consensus.cc:3032] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:58.786547 18076 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:58.783229 13661 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:58.787717 18413 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:58.820322 13807 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
W20260807 08:01:58.826277 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 81750us
I20260807 08:01:58.770898 18190 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:58.828478 18404 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:58.841485 18190 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.841849 18404 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:58.843804 18190 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.844204 18404 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:01:58.844426 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 61349.5us
I20260807 08:01:58.844806 18404 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:58.845144 13661 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:58.845258 18404 raft_consensus.cc:686] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:01:58.855659 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 70788.7us
I20260807 08:01:58.820765 18380 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:58.821489 18294 raft_consensus.cc:2781] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election won for term 7
W20260807 08:01:58.855295 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 88665.8us
I20260807 08:01:58.871181 13769 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:58.838323 18398 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:01:58.829710 18376 raft_consensus.cc:729] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:58.847105 18140 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:58.839592 17654 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:01:58.898514 13769 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:01:58.903118 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 486372us
W20260807 08:01:58.903731 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 56971.1us
I20260807 08:01:58.905997 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:58.906402 14583 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:58.879158 18398 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:58.913585 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:58.920902 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:58.921182 13811 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:58.892846 18140 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:58.891543 18113 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.922556 14580 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:58.891168 18259 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.890648 18291 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.895324 18294 raft_consensus.cc:686] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:01:58.934821 18405 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:58.935256 18405 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:58.935717 18405 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:58.935195 18294 consensus_queue.cc:237] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.924001 18398 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:58.937937 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:58.938354 18356 log.cc:924] Time spent T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.891s user 0.013s sys 0.013s
I20260807 08:01:58.939165 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:58.902451 18201 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:58.940662 18410 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:58.941108 18410 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:58.941471 18410 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:58.942008 17915 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.924130 18254 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:58.943102 13811 raft_consensus.cc:1261] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260807 08:01:58.943984 18376 consensus_queue.cc:260] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.923712 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:58.923206 17924 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:01:58.834193 18344 consensus_queue.cc:260] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:58.927572 18393 log.cc:924] Time spent T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.197s user 0.000s sys 0.001s
W20260807 08:01:58.929848 18394 log.cc:924] Time spent T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.186s user 0.001s sys 0.000s
I20260807 08:01:58.931195 18427 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:58.931777 18436 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:01:58.932777 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:58.933511 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:01:58.964579 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 118347us
I20260807 08:01:58.965699 13804 raft_consensus.cc:2396] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260807 08:01:58.934026 18259 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:58.900969 17654 raft_consensus.cc:686] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:58.897405 18230 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.829496 18399 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.958355 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:58.923969 18140 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:58.945225 18087 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:58.956118 17924 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:58.965032 18436 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:58.982208 13540 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:58.975299 17924 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Advancing to term 9
W20260807 08:01:58.987435 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 81518.4us
W20260807 08:01:58.986754 18090 consensus_meta.cc:287] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.656s user 0.011s sys 0.000s
I20260807 08:01:58.969673 18259 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.993436 18440 consensus_queue.cc:1035] T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260807 08:01:58.984485 18436 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:58.994980 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:59.003271 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:59.006168 13811 raft_consensus.cc:2396] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260807 08:01:59.006719 14626 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:59.010607 13540 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:58.968086 18404 consensus_queue.cc:237] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.019486 18419 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:59.019865 18428 raft_consensus.cc:2726] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:59.001590 17654 consensus_queue.cc:237] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:58.991968 18087 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:59.026021 14625 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
W20260807 08:01:59.032634 13769 leader_election.cc:334] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.810s (SENT)
I20260807 08:01:59.020040 18419 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:59.059017 13769 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:59.060739 13540 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.011724 18090 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:59.034740 18294 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:59.065970 18452 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:59.066538 18452 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.068015 18186 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:01:59.068622 18452 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:59.069720 18447 consensus_queue.cc:1035] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260807 08:01:59.071319 18445 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:59.071813 18445 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:59.072754 13580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1825) took 1620 ms (client timeout 1783 ms). Trace:
W20260807 08:01:59.073135 13580 rpcz_store.cc:269] 0807 08:01:57.452449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:57.603799 (+151350us) service_pool.cc:227] Handling call
0807 08:01:59.072538 (+1468739us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b540021f2d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:59.072673 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":634442624}
I20260807 08:01:59.061807 18254 raft_consensus.cc:2726] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:59.073616 17924 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:01:59.074364 18435 log.cc:924] Time spent T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260807 08:01:59.075692 13587 raft_consensus.cc:1226] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260807 08:01:59.076036 17924 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:59.077625 18399 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:59.088789 13591 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:01:59.086915 13540 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:59.080179 18367 consensus_meta.cc:287] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.530s user 0.008s sys 0.002s
I20260807 08:01:59.088390 18087 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:01:59.115046 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:59.108767 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:59.112743 13808 raft_consensus.cc:2396] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 08:01:59.121924 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:59.122150 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 89568.1us
I20260807 08:01:59.117285 13811 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
W20260807 08:01:59.108426 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 119805us
I20260807 08:01:59.082175 18294 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:59.124524 18410 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.124161 13661 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.126233 18294 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:59.126261 18076 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:01:59.126705 18410 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:59.113675 18308 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:59.127702 18076 consensus_queue.cc:1046] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:01:59.127547 18308 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.129307 18076 consensus_queue.cc:1046] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 08:01:59.130450 18230 raft_consensus.cc:3032] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 LEADER]: Stepping down as leader of term 9
I20260807 08:01:59.130817 18230 raft_consensus.cc:729] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:59.106503 18428 consensus_queue.cc:1046] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260807 08:01:59.131748 18230 consensus_queue.cc:260] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:59.132344 18428 raft_consensus.cc:3032] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:59.132690 18428 raft_consensus.cc:729] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:59.132918 18427 raft_consensus.cc:2726] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:59.133807 18230 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:01:59.112708 18398 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.136098 13817 raft_consensus.cc:1226] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:01:59.137171 18398 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:59.095738 18448 raft_consensus.cc:2726] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260807 08:01:59.137817 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 56225.5us
I20260807 08:01:59.150563 13540 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:59.102303 18367 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.113899 18090 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:59.133541 18428 consensus_queue.cc:260] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.141677 18445 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:59.110396 18259 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:59.112057 18140 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.162750 18140 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:59.157003 18367 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:59.157588 18417 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:01:59.158252 18308 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:59.109650 18419 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:59.158712 13769 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.159037 18381 consensus_queue.cc:1046] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260807 08:01:59.165390 14620 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 2 FOLLOWER]: Advancing to term 3
W20260807 08:01:59.167218 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 262839us
I20260807 08:01:59.170037 14580 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:59.153157 14626 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:01:59.170954 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:01:59.176561 13817 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:59.180441 13540 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.183290 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:59.195729 13540 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.184139 13805 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:01:59.197835 13769 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:59.187597 18428 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:59.200562 14583 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:01:59.201246 18360 log.cc:924] Time spent T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 1.159s user 0.007s sys 0.030s
I20260807 08:01:59.201776 14583 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.202617 18319 raft_consensus.cc:2726] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:59.202989 14583 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.203447 18319 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:01:59.203892 18319 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:59.204217 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.204305 18319 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:59.166353 18436 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:59.205714 18405 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.207291 18436 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:59.207716 18405 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:59.179387 18230 consensus_queue.cc:1035] T 23f6b5b24a8345d0ba918651b4a2c85f P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260807 08:01:59.210912 13804 raft_consensus.cc:1226] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: []
W20260807 08:01:59.197104 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 163375us
I20260807 08:01:59.172832 18087 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election won for term 9
W20260807 08:01:59.263866 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 77610.6us
I20260807 08:01:59.263892 13590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 649) took 1975 ms. Trace:
W20260807 08:01:59.283740 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 71853.3us
I20260807 08:01:59.186746 18201 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:01:59.325170 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
W20260807 08:01:59.325295 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 129634us
I20260807 08:01:59.196218 18445 raft_consensus.cc:2726] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:59.214430 14620 raft_consensus.cc:2454] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:01:59.284782 13769 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.329890 13540 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.330909 13808 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 8: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:59.331302 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:01:59.326951 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 129081us
I20260807 08:01:59.334460 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:59.265578 13590 rpcz_store.cc:276] 0807 08:01:57.288540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:57.288685 (+ 145us) service_pool.cc:227] Handling call
0807 08:01:57.308621 (+ 19936us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:57.347692 (+ 39071us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 08:01:57.347723 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:01:57.382052 (+ 34329us) log.cc:842] Serialized 26 byte log entry
0807 08:01:57.762714 (+380662us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:01:57.762785 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:57.762958 (+ 173us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:01:59.200315 (+1437357us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:01:59.210359 (+ 10044us) raft_consensus.cc:1671] finished
0807 08:01:59.210388 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:01:59.263808 (+ 53420us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3588,"raft.queue_time_us":16009,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":52826,"spinlock_wait_cycles":4585344,"thread_start_us":380266,"threads_started":1,"wal-append.queue_time_us":617514}
I20260807 08:01:59.332887 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:59.335129 13702 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 9.
W20260807 08:01:59.335572 18469 log.cc:924] Time spent T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
I20260807 08:01:59.336054 14620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42248 (request call id 753) took 1183 ms. Trace:
I20260807 08:01:59.336438 14620 rpcz_store.cc:276] 0807 08:01:58.150498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:58.150627 (+ 129us) service_pool.cc:227] Handling call
0807 08:01:59.164753 (+1014126us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b58005600b0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cee6f63 00007f6b1cf4aaba 00007f6b1cf623af 00007f6b1cf63b4e 00007f6b1cf6f228 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c
0807 08:01:59.333477 (+168724us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1907742080}
I20260807 08:01:59.337157 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:59.337958 13805 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:01:59.338064 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:59.337954 18446 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260807 08:01:59.338580 18438 log.cc:924] Time spent T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.207s user 0.002s sys 0.000s
I20260807 08:01:59.339205 18423 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:01:59.338670 13580 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:01:59.344480 13817 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:01:59.264521 18399 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:59.262501 18253 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:59.347992 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:59.186962 18345 consensus_queue.cc:1046] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.141s
I20260807 08:01:59.329074 18448 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:59.341225 18414 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:59.329376 18201 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:01:59.334805 18465 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:01:59.378589 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 48765.6us
W20260807 08:01:59.379540 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 163770us
I20260807 08:01:59.380136 13769 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:59.354694 18367 raft_consensus.cc:2781] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:01:59.368047 17654 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:59.367502 18222 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:59.379225 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:01:59.389051 13769 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.389791 18445 raft_consensus.cc:2741] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260807 08:01:59.347148 18427 consensus_queue.cc:1046] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
W20260807 08:01:59.422010 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 46170.1us
I20260807 08:01:59.376861 18201 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:59.370234 18087 raft_consensus.cc:686] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 9 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:01:59.438162 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 49211.3us
I20260807 08:01:59.439816 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893
W20260807 08:01:59.443826 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 84832.9us
I20260807 08:01:59.392993 13661 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.454568 14580 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
W20260807 08:01:59.454516 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 120251us
I20260807 08:01:59.368547 18399 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.424973 18090 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:01:59.465915 18407 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:59.386555 18367 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:01:59.474977 13540 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.476838 13805 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 8. Current term is 9. Ops: []
W20260807 08:01:59.528679 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 53810.2us
I20260807 08:01:59.429687 18446 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:59.456563 18294 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:01:59.449872 18413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:56576 (request call id 970) took 15740 ms. Trace:
I20260807 08:01:59.422427 18196 consensus_queue.cc:1035] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
W20260807 08:01:59.524840 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 50201.6us
W20260807 08:01:59.574404 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 183434us
W20260807 08:01:59.526204 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 71728.7us
I20260807 08:01:59.529716 13540 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.452607 13769 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:59.459893 18448 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:59.454622 17654 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:59.530601 18090 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:59.540923 18367 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:01:59.386684 18465 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.568850 18452 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:59.585089 18156 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:59.568061 18308 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260807 08:01:59.568360 18345 raft_consensus.cc:2781] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election won for term 8
W20260807 08:01:59.592474 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 139959us
I20260807 08:01:59.524765 18320 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260807 08:01:59.568190 18294 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:59.612155 18294 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:01:59.613090 18399 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:59.618938 18344 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:59.589732 18090 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:01:59.623306 18419 raft_consensus.cc:3032] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:01:59.585028 18087 consensus_queue.cc:237] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:01:59.625816 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 49837.5us
W20260807 08:01:59.602646 18481 log.cc:924] Time spent T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20260807 08:01:59.603621 18480 log.cc:924] Time spent T b09437796639470caac9548c71d72fb2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260807 08:01:59.603497 18268 consensus_queue.cc:1046] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:01:59.604475 18376 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:59.571034 18413 rpcz_store.cc:276] 0807 08:01:43.687405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:43.687566 (+ 161us) service_pool.cc:227] Handling call
0807 08:01:43.828217 (+140651us) log.cc:842] Serialized 501 byte log entry
0807 08:01:44.579422 (+751205us) spinlock_profiling.cc:243] Waited 4.81 ms on lock 0x7b54002c5c20. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf04416 00007f6b1cefddaa 00007f6b1cefd014 00007f6b1cee5545 00007f6b1ceea771 00007f6b1ceea6f9 00007f6b1ceea689 00007f6b1ceea651 00007f6b1cee981d 00007f6b232a7204
0807 08:01:45.629291 (+1049869us) spinlock_profiling.cc:243] Waited 5.85 ms on lock 0x7b54002c5c20. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf04416 00007f6b1cefddaa 00007f6b1cefd014 00007f6b1cee5545 00007f6b1ceea771 00007f6b1ceea6f9 00007f6b1ceea689 00007f6b1ceea651 00007f6b1cee981d 00007f6b232a7204
0807 08:01:59.417455 (+13788164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":19305,"raft.run_cpu_time_us":10253,"raft.run_wall_time_us":2282271,"spinlock_wait_cycles":24520192,"thread_start_us":116582,"threads_started":1,"wal-append.queue_time_us":225066,"wal-append.run_cpu_time_us":1972,"wal-append.run_wall_time_us":1026874}
I20260807 08:01:59.606992 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1945) took 2273 ms. Trace:
I20260807 08:01:59.653084 13587 rpcz_store.cc:276] 0807 08:01:57.333160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:57.705995 (+372835us) service_pool.cc:227] Handling call
0807 08:01:59.075437 (+1369442us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:01:59.076074 (+ 637us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:01:59.606807 (+530733us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540021f2d0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:59.606935 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3066529152}
I20260807 08:01:59.569205 18407 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.623718 18419 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:01:59.616529 13430 catalog_manager.cc:5495] T 9ab8c56441f84d86b5798c528463965c P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:59.587525 18446 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:59.650020 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:01:59.660904 13807 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:01:59.662045 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 132115us
I20260807 08:01:59.662915 13540 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.630658 18428 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:59.664716 13540 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.666185 13418 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b09437796639470caac9548c71d72fb2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 5)
I20260807 08:01:59.666023 13430 catalog_manager.cc:5495] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:59.670766 13540 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.651847 18345 raft_consensus.cc:686] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 8 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:01:59.585924 18448 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.676986 18320 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:01:59.677289 13430 catalog_manager.cc:5495] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 4 to 6, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68). New cstate: current_term: 6 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260807 08:01:59.678716 18472 log.cc:924] Time spent T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.333s user 0.002s sys 0.000s
I20260807 08:01:59.679733 13701 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:01:59.680743 18367 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:59.680611 13430 catalog_manager.cc:5495] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 5 to 6, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 6 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:01:59.681429 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1618) took 1312 ms. Trace:
I20260807 08:01:59.681717 18443 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:59.688795 13701 rpcz_store.cc:276] 0807 08:01:58.369450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:58.380464 (+ 11014us) service_pool.cc:227] Handling call
0807 08:01:59.681279 (+1300815us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54001d38d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:59.681376 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2985980288}
I20260807 08:01:59.663698 18308 raft_consensus.cc:3032] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 8 LEADER]: Stepping down as leader of term 8
I20260807 08:01:59.670113 18268 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260807 08:01:59.696599 13769 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.699599 13430 catalog_manager.cc:5495] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "22ff0a1f4a4241d794ab182432c85893" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260807 08:01:59.690308 18405 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:01:59.694363 18428 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:01:59.699396 18407 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:59.695020 17218 consensus_queue.cc:1035] T 70356515c14242539aa745f7e028f6b6 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260807 08:01:59.698163 18308 raft_consensus.cc:729] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:01:59.699864 18465 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:59.721923 13816 raft_consensus.cc:1226] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
W20260807 08:01:59.711129 18353 log.cc:924] Time spent T 0f7a6ca6100143da93759f88722f56e3 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 1.576s user 0.016s sys 0.008s
I20260807 08:01:59.719422 13769 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:59.733076 13769 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:01:59.694732 18320 consensus_queue.cc:1035] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.032s
W20260807 08:01:59.736724 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 84495.7us
I20260807 08:01:59.739377 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:59.696478 18446 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260807 08:01:59.717450 18206 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:01:59.743666 13816 raft_consensus.cc:1226] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:01:59.748323 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:59.749101 14620 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 8: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
W20260807 08:01:59.750131 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 357226us
I20260807 08:01:59.751462 13661 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.694628 18419 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:01:59.753396 18419 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:59.755420 14620 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260807 08:01:59.755738 14620 pending_rounds.cc:76] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Aborting all ops after (but not including) 7
I20260807 08:01:59.756009 14620 pending_rounds.cc:97] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260807 08:01:59.756094 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:59.756332 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:59.755748 13580 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 08:01:59.756768 13591 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:01:59.758942 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:59.759514 13592 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:01:59.760341 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:59.756942 13710 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:59.756268 14620 raft_consensus.cc:2866] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:01:59.766579 13661 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.773774 13661 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.737576 18398 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:01:59.757169 18445 consensus_queue.cc:1046] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260807 08:01:59.734398 18407 consensus_queue.cc:1046] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260807 08:01:59.781047 14583 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:01:59.745642 18374 raft_consensus.cc:2781] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:01:59.790433 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:59.790799 13661 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.783989 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:01:59.719954 18405 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.780092 18452 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:01:59.789746 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:59.734627 18345 consensus_queue.cc:237] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.798301 13661 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.730188 18090 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:01:59.774688 18156 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 08:01:59.813272 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 140707us
I20260807 08:01:59.815160 13661 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:01:59.780522 18140 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:59.820063 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:59.826428 13702 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:01:59.822343 13540 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.827582 13540 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:01:59.829125 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:01:59.837158 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:01:59.808921 18447 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:01:59.806685 18253 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 08:01:59.935531 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 106454us
I20260807 08:01:59.842726 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 609) took 1504 ms. Trace:
I20260807 08:01:59.946161 13584 raft_consensus.cc:1226] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 7. Current term is 8. Ops: []
I20260807 08:01:59.948729 13540 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.950242 13540 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:01:59.945940 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:01:59.792117 18337 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260807 08:01:59.939024 14630 rpcz_store.cc:276] 0807 08:01:58.338032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:58.341604 (+ 3572us) service_pool.cc:227] Handling call
0807 08:01:59.842547 (+1500943us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b5400522454. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:01:59.842666 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1753946496}
I20260807 08:01:59.974658 13708 raft_consensus.cc:3037] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:01:59.957201 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 199378us
I20260807 08:01:59.966440 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:01:59.996112 14629 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:59.963843 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:01:59.967382 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 124974us
I20260807 08:01:59.969241 14620 raft_consensus.cc:1261] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:59.968225 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:59.991043 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:00.000064 14622 raft_consensus.cc:2379] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:02:00.001725 14626 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:01:59.816782 17654 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:01:59.825963 18156 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:00.072054 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:00.072278 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.072927 13809 raft_consensus.cc:2396] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260807 08:02:00.062734 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.074445 14626 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 08:02:00.075523 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.076346 13590 raft_consensus.cc:2396] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260807 08:02:00.076474 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:00.077206 13579 raft_consensus.cc:2354] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8.
I20260807 08:01:59.828434 17885 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:02:00.013825 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:00.082005 13709 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:01:59.841642 18303 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 08:01:59.843704 18425 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:00.069484 13702 raft_consensus.cc:1261] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260807 08:01:59.993410 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 57882.6us
I20260807 08:01:59.946358 18230 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:00.069566 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.087517 13585 raft_consensus.cc:2454] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:02:00.006052 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:01:59.947065 18253 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:01:59.977114 18319 consensus_meta.cc:287] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.772s user 0.010s sys 0.003s
I20260807 08:02:00.092134 13700 raft_consensus.cc:1226] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 5. Current term is 6. Ops: []
I20260807 08:02:00.092245 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.092891 13589 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 9.
I20260807 08:02:00.080332 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:01:59.942164 18452 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:01:59.825759 18500 raft_consensus.cc:2726] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:01:59.935797 18448 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:01:59.956703 18206 raft_consensus.cc:686] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:00.087450 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.080190 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.100744 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.079085 18156 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:00.089596 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:01:59.838728 18222 consensus_queue.cc:1046] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.239s
I20260807 08:02:00.003232 18374 raft_consensus.cc:686] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:02:00.111200 18497 log.cc:924] Time spent T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.285s user 0.000s sys 0.000s
I20260807 08:02:00.005128 18268 consensus_queue.cc:1035] T 2acf2131e15e4fc9b2be438b9f3579b1 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20260807 08:02:00.079855 18293 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:01:59.946766 18440 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:01:59.989720 18315 raft_consensus.cc:2741] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 08:01:59.990561 17218 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:02:00.093007 18253 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:01:59.699980 18443 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.093726 13591 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:01:59.809330 18308 consensus_queue.cc:260] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.065313 18367 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:01:59.966624 18428 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:02:00.101926 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.132205 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.134696 13587 raft_consensus.cc:2379] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 6: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:00.132977 13592 raft_consensus.cc:2454] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:00.101178 18319 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.082773 18425 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.104315 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:00.104482 13703 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:02:00.104997 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:00.139989 14622 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:02:00.105798 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.138849 18380 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:00.121552 17218 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:00.141451 18380 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.142055 17218 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:02:00.142974 18344 consensus_meta.cc:287] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.524s user 0.004s sys 0.006s
I20260807 08:02:00.106565 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:00.143682 18380 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:00.144166 18380 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:00.144532 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.144608 18380 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.106766 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.107896 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.145792 13702 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:02:00.146433 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.147192 13585 raft_consensus.cc:2379] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 4: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:00.080744 18447 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:00.147977 18447 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.150007 18447 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:00.110394 18493 consensus_queue.cc:1046] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.175s
I20260807 08:02:00.084538 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:00.150895 14619 raft_consensus.cc:2379] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:02:00.108703 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:00.151858 13705 raft_consensus.cc:2454] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 9.
I20260807 08:02:00.155067 17218 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.156566 13580 raft_consensus.cc:2379] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 6: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:00.156567 18380 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:00.157673 13709 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:00.158241 17218 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:00.119298 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.094031 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:00.124003 18485 consensus_queue.cc:1035] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.054s
I20260807 08:02:00.162822 13700 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:02:00.124346 18090 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:00.124698 18478 consensus_queue.cc:1046] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260807 08:01:59.740959 18201 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:00.138173 18294 consensus_meta.cc:287] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.525s user 0.010s sys 0.003s
I20260807 08:02:00.126482 18440 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.119938 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.194422 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.194658 18440 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:00.123867 18206 consensus_queue.cc:237] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.178047 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.168473 13590 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:00.069444 18398 raft_consensus.cc:686] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:00.097018 18230 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.190958 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.191323 14625 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260807 08:02:00.166481 14629 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:02:00.134840 18428 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:00.195444 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.254647 13585 raft_consensus.cc:2379] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 7: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:00.190871 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.161615 18308 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:00.117818 18370 consensus_queue.cc:1046] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20260807 08:02:00.186029 13769 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:00.240865 13814 raft_consensus.cc:1226] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:02:00.243094 13591 raft_consensus.cc:2454] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3.
I20260807 08:02:00.240137 18294 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.264290 18344 consensus_queue.cc:1046] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:02:00.265095 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:00.111912 18374 consensus_queue.cc:237] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.265403 13805 raft_consensus.cc:1226] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 5. Current term is 6. Ops: []
I20260807 08:02:00.265884 18294 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:00.177757 14630 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:55594
I20260807 08:02:00.257823 18428 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:00.268713 14630 raft_consensus.cc:595] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:00.269254 14630 raft_consensus.cc:603] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:02:00.269842 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:00.271198 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.140159 18367 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:00.271843 13585 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 9.
I20260807 08:02:00.272110 18367 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.134652 18293 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.277089 18293 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:00.278774 18201 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:00.280917 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 95010.9us
I20260807 08:02:00.281492 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:00.249208 14627 raft_consensus.cc:1226] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260807 08:02:00.251995 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.255535 13587 raft_consensus.cc:2454] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:02:00.258430 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.253326 13703 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:02:00.190949 18445 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:00.317633 14630 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:55594
I20260807 08:02:00.295586 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:00.295472 13661 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:00.318325 13814 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 9
I20260807 08:02:00.303434 13711 raft_consensus.cc:1261] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260807 08:02:00.318284 14630 raft_consensus.cc:595] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:00.282172 13769 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:00.320255 14630 raft_consensus.cc:603] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:02:00.303701 13807 raft_consensus.cc:3032] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:02:00.320744 13807 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:00.306159 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:00.322217 13590 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:02:00.295048 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:00.286084 13588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56576 (request call id 1029) took 1396 ms. Trace:
I20260807 08:02:00.327508 13585 raft_consensus.cc:1226] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 3. Current term is 4. Ops: []
I20260807 08:02:00.331051 13769 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:02:00.311084 13710 consensus_meta.cc:287] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.520s user 0.008s sys 0.000s
I20260807 08:02:00.326122 13588 rpcz_store.cc:276] 0807 08:01:58.890036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:58.890181 (+ 145us) service_pool.cc:227] Handling call
0807 08:02:00.285253 (+1395072us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001e4e54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0af0 00007f6b1cf73f2a 00007f6b21eb0c6e 00007f6b1b5a7807 00007f6b1b5a7792 00007f6b1b5a76c1 00007f6b1b5a763c 00007f6b1b5a68f2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:00.285984 (+ 731us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2732817280}
W20260807 08:02:00.318415 18414 consensus_meta.cc:287] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.942s user 0.010s sys 0.001s
I20260807 08:02:00.301047 18432 raft_consensus.cc:3032] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
I20260807 08:02:00.300148 18403 raft_consensus.cc:3032] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:02:00.328644 18512 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260807 08:02:00.300324 18440 raft_consensus.cc:3032] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:02:00.326989 18294 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:02:00.299286 18156 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.295256 18168 raft_consensus.cc:3032] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:02:00.359977 18402 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260807 08:02:00.361742 18414 raft_consensus.cc:2726] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:00.361774 13807 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.298051 18478 raft_consensus.cc:3032] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 LEADER]: Stepping down as leader of term 8
I20260807 08:02:00.286998 18425 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:00.359615 13661 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:00.337488 13585 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:00.282419 18516 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 08:02:00.294015 18405 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:00.371627 13591 raft_consensus.cc:3037] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:00.374694 13661 leader_election.cc:302] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:00.332686 13710 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 9.
I20260807 08:02:00.303021 18230 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:00.296622 18319 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:00.426272 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:00.428014 13588 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260807 08:02:00.300218 18419 consensus_meta.cc:287] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.546s user 0.008s sys 0.000s
I20260807 08:02:00.428990 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.435659 13585 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:02:00.436712 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.437400 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.366127 18440 raft_consensus.cc:729] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:00.386152 13702 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 8. Current term is 9. Ops: [6.3-8.4]
I20260807 08:02:00.363279 13708 raft_consensus.cc:2454] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
W20260807 08:02:00.366477 18376 consensus_meta.cc:287] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.711s user 0.010s sys 0.000s
I20260807 08:02:00.365796 18512 raft_consensus.cc:2726] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260807 08:02:00.373750 18478 raft_consensus.cc:729] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:00.363320 18432 raft_consensus.cc:729] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:00.456019 13589 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:00.326261 18493 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:00.369493 18168 raft_consensus.cc:729] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:00.377166 18516 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:00.366760 18294 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:00.369292 18156 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:02:00.497784 13769 leader_election.cc:334] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.506s (SENT)
I20260807 08:02:00.556432 13769 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:00.494451 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.429682 18293 raft_consensus.cc:2781] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:02:00.429885 18410 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:00.387457 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:00.326119 18445 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.387256 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:00.489064 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.335561 18381 consensus_queue.cc:1035] T c02eb1166e17491f82cfbd9c5cd030a2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.121s
I20260807 08:02:00.521966 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:00.472496 13591 raft_consensus.cc:2454] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:02:00.535099 13661 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:00.542071 13814 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 9.
I20260807 08:02:00.559700 13585 raft_consensus.cc:2379] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:00.571998 14580 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:00.442085 18402 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 08:02:00.561658 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 63918.9us
I20260807 08:02:00.562618 14630 raft_consensus.cc:2396] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:02:00.363699 18403 raft_consensus.cc:729] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:00.336036 18446 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:00.572903 13661 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:02:00.573706 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 490216us
W20260807 08:02:00.570343 18482 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.817s user 0.005s sys 0.020s
I20260807 08:02:00.364363 18253 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.522486 18319 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:02:00.581774 13769 leader_election.cc:334] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 2.192s (SENT)
I20260807 08:02:00.556041 18294 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:00.582695 14618 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:55594
I20260807 08:02:00.579694 13585 raft_consensus.cc:1226] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:02:00.544211 18268 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:00.371682 18398 consensus_queue.cc:237] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.555856 18516 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:00.586681 13661 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:00.563158 18522 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:00.377861 18518 consensus_queue.cc:1035] T cc32413495ee41909c7c5494c8095084 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.207s
I20260807 08:02:00.596815 18168 consensus_queue.cc:260] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.598253 18168 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:01:59.979473 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 137273us
I20260807 08:02:00.592777 18446 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.601720 18432 consensus_queue.cc:260] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.603812 18446 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:00.563846 18440 consensus_queue.cc:260] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.605265 18440 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:02:00.611997 18303 raft_consensus.cc:3032] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:02:00.612406 18303 raft_consensus.cc:729] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:00.618935 13809 raft_consensus.cc:1226] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:02:00.620452 18378 consensus_queue.cc:1046] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260807 08:02:00.621199 18461 consensus_queue.cc:1159] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "77da774f2bb54cf28302d53d0db224ad" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 3 index: 3 } }
I20260807 08:02:00.623754 18443 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:00.624800 18253 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:00.588290 14618 raft_consensus.cc:595] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:00.568207 18293 raft_consensus.cc:686] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:02:00.592367 13769 leader_election.cc:334] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 2.397s (SENT)
I20260807 08:02:00.568370 18410 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.586759 18367 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:00.587162 18405 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 08:02:00.588984 18419 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 08:02:00.596410 18403 consensus_queue.cc:260] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.605312 14626 raft_consensus.cc:1226] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9. Ops: []
I20260807 08:02:00.595240 13661 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:00.626634 13807 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:02:00.646076 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 50922.1us
I20260807 08:02:00.647049 13661 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:02:00.585709 18482 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.833s user 0.005s sys 0.020s
I20260807 08:02:00.582383 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:00.651392 13706 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:02:00.655344 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:00.656070 13809 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:00.660597 13661 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:02:00.662674 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 70328.4us
I20260807 08:02:00.664290 13814 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:00.594661 18485 consensus_queue.cc:1035] T 38e7d8fef17f4bdb92d0c21739339421 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260807 08:02:00.595610 18319 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:00.667732 13540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.008s sys 0.019s
W20260807 08:02:00.732201 13540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.656s user 0.008s sys 0.019s
W20260807 08:02:00.709784 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 49260.1us
I20260807 08:02:00.592391 18268 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.657579 17943 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:02:00.728875 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 64319.1us
W20260807 08:02:00.734505 13430 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.018s user 0.010s sys 0.025s
W20260807 08:02:00.734877 13430 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.019s user 0.010s sys 0.025s
I20260807 08:02:00.734805 14583 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:00.736017 14583 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:00.730023 18308 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:00.652983 18443 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 08:02:00.659515 18253 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:00.806421 18344 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:00.806737 18253 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:00.807276 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 73846.3us
I20260807 08:02:00.808542 13661 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:00.808775 18253 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:00.809072 18319 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:00.657730 18405 raft_consensus.cc:686] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 9 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
W20260807 08:02:00.810089 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 67959.8us
I20260807 08:02:00.810849 14629 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:02:00.811412 13807 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260807 08:02:00.811771 13807 pending_rounds.cc:76] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 7
I20260807 08:02:00.812043 13807 pending_rounds.cc:97] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260807 08:02:00.812323 13807 raft_consensus.cc:2866] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:02:00.613224 18303 consensus_queue.cc:260] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:00.800638 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:00.802006 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:02:00.803303 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 67338.7us
I20260807 08:02:00.823125 14583 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:00.657764 18428 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:00.800230 17681 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:00.833551 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 730) took 1120 ms. Trace:
I20260807 08:02:00.817315 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 724) took 1238 ms. Trace:
I20260807 08:02:00.830447 13540 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:02:00.672573 18436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.016s sys 0.010s
I20260807 08:02:00.824506 14583 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:00.827272 13661 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:00.829634 13585 raft_consensus.cc:1261] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260807 08:02:00.750352 13769 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.675s (SENT) [suppressed 19 similar messages]
I20260807 08:02:00.831527 13591 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2"
dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:39822
I20260807 08:02:00.822190 14618 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
W20260807 08:02:00.837266 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 86958.1us
I20260807 08:02:00.838671 13769 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:00.839087 14583 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:00.833984 13807 rpcz_store.cc:276] 0807 08:01:59.712704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:59.712825 (+ 121us) service_pool.cc:227] Handling call
0807 08:01:59.713134 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:00.303610 (+590476us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54002f3620. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cefb730 00007f6b1cf6d300 00007f6b1cf6e371 00007f6b1cf6aedf 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc
0807 08:02:00.626136 (+322526us) spinlock_profiling.cc:243] Waited 8 ms on lock 0x7b58005924b0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cee6f63 00007f6b1cf4aaba 00007f6b1cf623af 00007f6b1cf63b4e 00007f6b1cf6dc89 00007f6b1cf6e556 00007f6b1cf6aedf 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684
0807 08:02:00.833273 (+207137us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:00.833478 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":860335360}
I20260807 08:02:00.840345 14583 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:00.731696 18230 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:00.834338 13805 rpcz_store.cc:276] 0807 08:01:59.578974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:59.579124 (+ 150us) service_pool.cc:227] Handling call
0807 08:01:59.579499 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:00.265823 (+686324us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:00.817014 (+551191us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b540027bad4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:00.817231 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1577024896}
I20260807 08:02:00.596475 18478 consensus_queue.cc:260] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:00.843781 13769 leader_election.cc:334] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.793s (SENT)
I20260807 08:02:00.853427 13769 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:00.810029 18414 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:00.848631 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.877085 13585 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:02:00.819518 18446 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:00.855342 18344 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:00.810456 18443 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:00.892848 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.857347 18319 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260807 08:02:00.858603 18523 log.cc:924] Time spent T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20260807 08:02:00.858484 13661 leader_election.cc:302] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:00.843889 18378 raft_consensus.cc:2726] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:02:00.859809 18419 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:02:00.841058 18436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.079s user 0.017s sys 0.010s
I20260807 08:02:00.833446 17654 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260807 08:02:00.865047 13769 leader_election.cc:334] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.815s (SENT)
I20260807 08:02:00.956825 13769 leader_election.cc:302] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:00.865701 18521 raft_consensus.cc:2726] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:00.867914 14623 raft_consensus.cc:1226] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:02:00.801836 18367 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:00.847777 18206 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:00.846905 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.809762 18156 consensus_queue.cc:1046] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 08:02:00.889371 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:02:00.852561 18515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260807 08:02:00.899857 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:00.905682 13585 raft_consensus.cc:2379] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 9: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:00.906136 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:00.836599 18425 raft_consensus.cc:2726] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:00.911357 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.913975 13710 raft_consensus.cc:1226] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 8. Current term is 9. Ops: []
I20260807 08:02:00.809945 18308 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:00.915132 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 56733.9us
W20260807 08:02:01.013118 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 148113us
I20260807 08:02:01.014058 13661 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:00.859335 18370 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:00.906124 17681 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:00.933780 17915 raft_consensus.cc:2726] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 08:02:00.873591 18445 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:00.875346 18489 raft_consensus.cc:2781] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 08:02:00.952111 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:01.010736 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:00.837002 18140 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.078s
I20260807 08:02:00.884344 18090 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:00.892733 18344 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:00.906688 18443 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:00.940196 18478 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:01.006202 13588 raft_consensus.cc:2454] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:02:01.008134 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:00.819994 18293 consensus_queue.cc:237] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:01.014506 13769 leader_election.cc:334] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.917s (SENT)
I20260807 08:02:01.076062 13769 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:00.956719 17654 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:00.885393 18414 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.024916 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:01.086993 13708 raft_consensus.cc:2379] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 7: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:01.086920 13657 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:02:01.007522 18523 log.cc:924] Time spent T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260807 08:02:01.030721 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:00.907459 18446 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.031441 18425 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:01.047248 18489 raft_consensus.cc:482] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:01.008033 18206 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:01.012260 18521 raft_consensus.cc:2726] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:00.958918 18419 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.065310 13592 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:01.012564 18410 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:01.079557 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:01.048702 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:00.980381 18405 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.006937 18253 consensus_queue.cc:1046] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
W20260807 08:02:01.090826 18534 log.cc:924] Time spent T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.152s user 0.000s sys 0.000s
W20260807 08:02:01.095424 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 80962.9us
W20260807 08:02:01.127862 13709 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.011s sys 0.013s
W20260807 08:02:01.128216 13709 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.535s user 0.011s sys 0.013s
I20260807 08:02:01.128504 13769 leader_election.cc:302] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:01.128696 13711 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 10.
I20260807 08:02:01.129549 13769 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:01.095865 18512 raft_consensus.cc:2726] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:02:01.053817 18427 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:02:01.131042 13701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.842s user 0.000s sys 0.025s
I20260807 08:02:01.109915 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:01.100889 18206 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:01.113457 13585 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:01.134859 13769 leader_election.cc:302] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:02:01.076589 18445 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:01.087716 17654 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:01.017858 18367 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.119853 13579 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 9.
I20260807 08:02:01.121214 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:01.099900 18489 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:01.095449 18446 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:02:01.131385 13701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.843s user 0.000s sys 0.025s
I20260807 08:02:01.074862 18344 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:02:01.134851 13700 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.013s sys 0.013s
W20260807 08:02:01.173553 13700 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.578s user 0.013s sys 0.013s
I20260807 08:02:01.174516 13700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1636) took 1087 ms. Trace:
I20260807 08:02:01.100924 18156 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 08:02:01.146122 18428 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:02:01.199051 13580 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.008s sys 0.033s
I20260807 08:02:01.108708 18381 consensus_queue.cc:1035] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260807 08:02:01.099646 18425 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.164986 13592 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 9.
I20260807 08:02:01.147931 18542 raft_consensus.cc:2781] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election won for term 8
W20260807 08:02:01.170697 18398 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.011s sys 0.035s
W20260807 08:02:01.141029 18345 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.840s user 0.002s sys 0.026s
I20260807 08:02:01.175585 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1849) took 1424 ms. Trace:
I20260807 08:02:01.140627 18521 raft_consensus.cc:2781] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election won for term 7
W20260807 08:02:01.074880 18423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s user 0.011s sys 0.054s
I20260807 08:02:01.140890 18436 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader election won for term 9
W20260807 08:02:01.188997 13809 consensus_meta.cc:287] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.532s user 0.006s sys 0.000s
I20260807 08:02:01.251180 13809 raft_consensus.cc:2396] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 08:02:01.164752 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:01.201998 13585 raft_consensus.cc:2454] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 9.
W20260807 08:02:01.201498 13580 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s user 0.008s sys 0.033s
I20260807 08:02:01.204077 13710 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 9: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
W20260807 08:02:01.209579 18533 log.cc:924] Time spent T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.181s user 0.001s sys 0.000s
I20260807 08:02:01.131165 17585 raft_consensus.cc:2726] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 08:02:01.248594 18345 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.947s user 0.003s sys 0.027s
I20260807 08:02:01.159544 18512 consensus_queue.cc:1046] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.043s
I20260807 08:02:01.174897 13700 rpcz_store.cc:276] 0807 08:02:00.096987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.191700 (+ 94713us) service_pool.cc:227] Handling call
0807 08:02:00.249926 (+ 58226us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:00.250154 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 08:02:00.279953 (+ 29799us) log.cc:842] Serialized 14 byte log entry
0807 08:02:01.173954 (+894001us) log.cc:842] Serialized 14 byte log entry
0807 08:02:01.174140 (+ 186us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:02:01.174188 (+ 48us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:02:01.174236 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.174331 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.174459 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30056,"prepare.queue_time_us":53126,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":5974,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5779,"thread_start_us":865109,"threads_started":2,"wal-append.queue_time_us":679266}
I20260807 08:02:01.266410 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:01.267709 18432 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:02:01.267877 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:01.270525 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:01.272157 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:02:01.216938 13590 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.016s sys 0.028s
W20260807 08:02:01.272903 13590 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.665s user 0.016s sys 0.028s
I20260807 08:02:01.141078 18427 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:01.255087 18543 log.cc:924] Time spent T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.247s user 0.000s sys 0.001s
I20260807 08:02:01.151809 18445 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.211493 18425 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:01.257594 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 427194us
I20260807 08:02:01.175781 18493 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260807 08:02:01.201246 18294 consensus_meta.cc:287] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.591s user 0.009s sys 0.000s
I20260807 08:02:01.340489 13540 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
W20260807 08:02:01.340498 13537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.000s sys 0.032s
W20260807 08:02:01.345290 13537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.514s user 0.000s sys 0.032s
W20260807 08:02:01.345692 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 514806us
I20260807 08:02:01.221796 13701 rpcz_store.cc:276] 0807 08:01:59.746848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:59.746990 (+ 142us) service_pool.cc:227] Handling call
0807 08:01:59.747253 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:01:59.747441 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 6
0807 08:01:59.833329 (+ 85888us) log.cc:842] Serialized 14 byte log entry
0807 08:02:00.089112 (+255783us) log.cc:842] Serialized 14 byte log entry
0807 08:02:00.288384 (+199272us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b1800195700. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b19fba4ca 00007f6b2369bf99 00007f6b19e7e990 00007f6b1a040864 00007f6b1a040155 00007f6b1cf633d0 00007f6b1cf75576 00007f6b1cf860d7 00007f6b1cf86058
0807 08:02:00.290464 (+ 2080us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b1800195700. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043764 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:02:01.170896 (+880432us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:02:01.170959 (+ 63us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:02:01.171017 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.171087 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.171218 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23304,"prepare.queue_time_us":344773,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":79,"raft.queue_time_us":220336,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":4774,"spinlock_wait_cycles":36982272,"thread_start_us":1168029,"threads_started":3,"wal-append.queue_time_us":305064}
W20260807 08:02:01.406639 14622 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.017s sys 0.024s
W20260807 08:02:01.406904 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 60347.9us
W20260807 08:02:01.446619 14582 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.021s sys 0.032s
W20260807 08:02:01.446969 14582 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.639s user 0.022s sys 0.032s
W20260807 08:02:01.448428 14622 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s user 0.018s sys 0.024s
W20260807 08:02:01.364603 18549 log.cc:924] Time spent T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.276s user 0.000s sys 0.001s
I20260807 08:02:01.273906 18446 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
W20260807 08:02:01.405052 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 58610.7us
I20260807 08:02:01.408855 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1854) took 1422 ms. Trace:
I20260807 08:02:01.452631 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1635) took 1370 ms. Trace:
I20260807 08:02:01.452481 13589 rpcz_store.cc:276] 0807 08:01:59.986368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.194183 (+207815us) service_pool.cc:227] Handling call
0807 08:02:01.408646 (+1214463us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b5400369554. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:01.408788 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":600123136}
W20260807 08:02:01.285836 18547 log.cc:924] Time spent T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.205s user 0.000s sys 0.001s
I20260807 08:02:01.276666 18076 raft_consensus.cc:3032] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:02:01.262183 18436 raft_consensus.cc:686] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:01.220403 18308 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:01.219200 18542 raft_consensus.cc:686] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:01.487548 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1632) took 1414 ms. Trace:
I20260807 08:02:01.488044 13706 rpcz_store.cc:276] 0807 08:02:00.073116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.091014 (+ 17898us) service_pool.cc:227] Handling call
0807 08:02:01.479391 (+1388377us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54001dced4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:01.487479 (+ 8088us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":710654336}
I20260807 08:02:01.488149 18542 consensus_queue.cc:237] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:01.490302 18516 consensus_meta.cc:287] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.875s user 0.009s sys 0.002s
I20260807 08:02:01.490429 14625 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 758) took 1737 ms. Trace:
W20260807 08:02:01.346969 18398 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s user 0.011s sys 0.036s
I20260807 08:02:01.490906 14625 rpcz_store.cc:276] 0807 08:01:59.753062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:59.766419 (+ 13357us) service_pool.cc:227] Handling call
0807 08:01:59.766711 (+ 292us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:00.253799 (+487088us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.490183 (+1236384us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b54005212d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:01.490370 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":975978240}
W20260807 08:02:01.340250 18482 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.011s sys 0.025s
I20260807 08:02:01.364322 18512 raft_consensus.cc:3032] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 08:02:01.451418 18201 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:01.282128 18378 raft_consensus.cc:3032] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Stepping down as leader of term 5
W20260807 08:02:01.406471 14627 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.005s sys 0.041s
I20260807 08:02:01.406459 18419 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:01.431805 18533 log.cc:924] Time spent T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260807 08:02:01.366904 18414 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:01.371490 18413 raft_consensus.cc:2726] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:02:01.453187 13709 rpcz_store.cc:276] 0807 08:02:00.081903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.159055 (+ 77152us) service_pool.cc:227] Handling call
0807 08:02:00.159640 (+ 585us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:00.298270 (+138630us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:00.298300 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:00.298708 (+ 408us) log.cc:842] Serialized 26 byte log entry
0807 08:02:00.563115 (+264407us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:02:01.128581 (+565466us) log.cc:842] Serialized 14 byte log entry
0807 08:02:01.128782 (+ 201us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.128858 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:01.452394 (+323536us) raft_consensus.cc:1671] finished
0807 08:02:01.452416 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.452562 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":141934,"prepare.queue_time_us":33581,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":141245,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":13937,"thread_start_us":798999,"threads_started":2,"wal-append.queue_time_us":583668}
W20260807 08:02:01.536368 14583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.014s sys 0.017s
W20260807 08:02:01.452112 18548 log.cc:924] Time spent T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.322s user 0.000s sys 0.001s
I20260807 08:02:01.480247 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 641) took 1403 ms. Trace:
W20260807 08:02:01.453967 18440 consensus_meta.cc:287] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.848s user 0.012s sys 0.000s
I20260807 08:02:01.483249 18308 consensus_queue.cc:1046] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260807 08:02:01.459015 18529 log.cc:924] Time spent T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.333s user 0.000s sys 0.001s
I20260807 08:02:01.368528 18294 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.459287 18076 raft_consensus.cc:729] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:02:01.527603 14627 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s user 0.005s sys 0.041s
I20260807 08:02:01.460379 18446 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:01.526859 13587 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.031s sys 0.047s
W20260807 08:02:01.523517 14580 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.000s sys 0.034s
W20260807 08:02:01.571643 14580 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.692s user 0.001s sys 0.034s
W20260807 08:02:01.571957 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 708938us
I20260807 08:02:01.520412 18403 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:02:01.534039 14618 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.016s sys 0.045s
I20260807 08:02:01.281371 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:01.544677 18413 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:01.538564 18201 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.546133 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1637) took 1436 ms. Trace:
I20260807 08:02:01.490736 18516 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.543064 14626 rpcz_store.cc:276] 0807 08:02:00.077016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.084347 (+ 7331us) service_pool.cc:227] Handling call
0807 08:02:00.084676 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:00.644898 (+560222us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.479992 (+835094us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b54004b32d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:01.480181 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1196946688}
W20260807 08:02:01.544068 14583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.664s user 0.014s sys 0.018s
I20260807 08:02:01.523674 18512 raft_consensus.cc:729] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:01.535144 18378 raft_consensus.cc:729] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
W20260807 08:02:01.571316 13587 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.911s user 0.032s sys 0.047s
I20260807 08:02:01.602540 13587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1972) took 1441 ms. Trace:
I20260807 08:02:01.603034 13587 rpcz_store.cc:276] 0807 08:02:00.161075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.316716 (+155641us) service_pool.cc:227] Handling call
0807 08:02:00.341116 (+ 24400us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:00.341296 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 8
0807 08:02:00.660217 (+318921us) log.cc:842] Serialized 14 byte log entry
0807 08:02:01.602128 (+941911us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:02:01.602180 (+ 52us) raft_consensus.cc:1631] Marking committed up to 8
0807 08:02:01.602232 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.602334 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.602472 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":738901,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"spinlock_wait_cycles":62720,"thread_start_us":1235400,"threads_started":2}
W20260807 08:02:01.604351 13584 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.725s user 0.007s sys 0.013s
W20260807 08:02:01.604669 13584 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.727s user 0.007s sys 0.013s
I20260807 08:02:01.539043 18206 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:01.522857 18482 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.858s user 0.011s sys 0.029s
W20260807 08:02:01.577832 14618 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.684s user 0.017s sys 0.045s
W20260807 08:02:01.604923 13807 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.007s sys 0.014s
I20260807 08:02:01.585502 13705 rpcz_store.cc:276] 0807 08:02:00.109991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.271023 (+161032us) service_pool.cc:227] Handling call
0807 08:02:00.271419 (+ 396us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:00.314792 (+ 43373us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:00.314836 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:00.315284 (+ 448us) log.cc:842] Serialized 26 byte log entry
0807 08:02:00.664917 (+349633us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:02:00.729581 (+ 64664us) log.cc:842] Serialized 14 byte log entry
0807 08:02:00.729788 (+ 207us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:00.729857 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:01.545888 (+816031us) raft_consensus.cc:1671] finished
0807 08:02:01.545918 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.546067 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90840,"prepare.queue_time_us":65839,"raft.queue_time_us":47031,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":7439,"spinlock_wait_cycles":6598528,"thread_start_us":231248,"threads_started":1,"wal-append.queue_time_us":726062}
I20260807 08:02:01.574996 18113 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:01.612784 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:02:01.614557 13661 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.012s sys 0.022s
W20260807 08:02:01.616590 13661 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.517s user 0.012s sys 0.022s
W20260807 08:02:01.617012 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 517402us
I20260807 08:02:01.617962 13661 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:01.616600 14626 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
W20260807 08:02:01.611060 13807 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.738s user 0.007s sys 0.014s
I20260807 08:02:01.538899 18344 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:01.667620 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 48260.8us
W20260807 08:02:01.668059 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 74563.7us
I20260807 08:02:01.567335 18414 consensus_queue.cc:1046] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260807 08:02:01.576277 18076 consensus_queue.cc:260] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.714198 18294 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:01.675398 14626 raft_consensus.cc:1261] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:02:01.676765 18482 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:01.602375 18367 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:02:01.678320 17943 consensus_meta.cc:287] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.860s user 0.014s sys 0.001s
I20260807 08:02:01.678421 18577 raft_consensus.cc:3032] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Stepping down as leader of term 7
W20260807 08:02:01.679428 18566 log.cc:924] Time spent T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260807 08:02:01.602311 14619 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:02:01.681260 14622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 647) took 1152 ms. Trace:
W20260807 08:02:01.681471 18443 consensus_meta.cc:287] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.602s user 0.006s sys 0.016s
I20260807 08:02:01.610260 18436 consensus_queue.cc:237] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.682574 18381 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:01.683204 18446 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:01.674599 18268 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:01.683252 17681 consensus_queue.cc:1046] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 08:02:01.685504 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1968) took 1580 ms. Trace:
I20260807 08:02:01.686712 18206 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:01.745766 13580 rpcz_store.cc:276] 0807 08:02:00.105353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.603546 (+498193us) service_pool.cc:227] Handling call
0807 08:02:00.603879 (+ 333us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:00.604229 (+ 350us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:00.604259 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:00.604680 (+ 421us) log.cc:842] Serialized 26 byte log entry
0807 08:02:01.252283 (+647603us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:02:01.254162 (+ 1879us) log.cc:842] Serialized 14 byte log entry
0807 08:02:01.254349 (+ 187us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.254424 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:01.478411 (+223987us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b18002e82e0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:02:01.685124 (+206713us) raft_consensus.cc:1671] finished
0807 08:02:01.685262 (+ 138us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.685436 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2172,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":5364,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":173,"spinlock_wait_cycles":6442240,"thread_start_us":647267,"threads_started":1,"wal-append.queue_time_us":954178}
W20260807 08:02:01.687706 18565 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20260807 08:02:01.748472 18493 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:01.748955 18493 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:01.688441 18344 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:01.689164 18516 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:01.612303 18496 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:01.608943 18544 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:01.694670 14583 leader_election.cc:302] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:01.696201 18345 raft_consensus.cc:979] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893: : Instructing follower 7a889001d6494fe5b7cbbe7da1dae302 to start an election
I20260807 08:02:01.696580 14627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 761) took 1026 ms. Trace:
I20260807 08:02:01.696852 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:02:01.743163 14619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 645) took 1262 ms (client timeout 1607 ms). Trace:
I20260807 08:02:01.780691 13705 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:01.697185 18518 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:02:01.780915 14619 rpcz_store.cc:269] 0807 08:02:00.477480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.533958 (+ 56478us) service_pool.cc:227] Handling call
0807 08:02:01.281206 (+747248us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b5800005198. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:02:01.739457 (+458251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5979904}
I20260807 08:02:01.781754 18518 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:01.783216 17654 consensus_meta.cc:287] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.618s user 0.012s sys 0.000s
I20260807 08:02:01.783725 17654 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.784258 18518 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:01.697469 18398 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:01.785254 18398 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.787678 18398 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:01.788355 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 93749us
I20260807 08:02:01.788429 18544 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:01.789135 18544 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:01.614341 18445 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:01.780360 14627 rpcz_store.cc:276] 0807 08:02:00.669904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.670135 (+ 231us) service_pool.cc:227] Handling call
0807 08:02:00.670514 (+ 379us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:00.670894 (+ 380us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:00.670932 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:00.682218 (+ 11286us) log.cc:842] Serialized 26 byte log entry
0807 08:02:01.570723 (+888505us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:02:01.570770 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.570872 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:01.696360 (+125488us) raft_consensus.cc:1671] finished
0807 08:02:01.696377 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.696514 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2358,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":66622,"thread_start_us":715499,"threads_started":1,"wal-append.queue_time_us":785992}
I20260807 08:02:01.701778 18380 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:01.697469 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:01.669929 18419 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:01.703285 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:01.723424 13591 raft_consensus.cc:595] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:01.592535 18413 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:01.689180 18489 consensus_meta.cc:287] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.509s user 0.008s sys 0.001s
I20260807 08:02:01.724063 13590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1973) took 1562 ms. Trace:
I20260807 08:02:01.799000 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:01.805665 13590 rpcz_store.cc:276] 0807 08:02:00.161374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.323044 (+161670us) service_pool.cc:227] Handling call
0807 08:02:00.607139 (+284095us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b5800001898. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00bbce 000000000034aea9 00007f6b21e37a30 00007f6b21efd7b4 00007f6b21efd821 00007f6b21ec12b0 00007f6b21eac7f9 00007f6b21eac2a2 00007f6b1b592684
0807 08:02:00.607284 (+ 145us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:00.607492 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 08:02:01.275104 (+667612us) log.cc:842] Serialized 14 byte log entry
0807 08:02:01.723076 (+447972us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:02:01.723148 (+ 72us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:02:01.723220 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.723836 (+ 616us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.723991 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2854,"prepare.queue_time_us":518581,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":7169,"raft.queue_time_us":6531,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":13450,"spinlock_wait_cycles":2884864,"thread_start_us":1110144,"threads_started":2}
W20260807 08:02:01.729552 13588 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.006s sys 0.039s
I20260807 08:02:01.730762 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1634) took 1656 ms. Trace:
I20260807 08:02:01.732924 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:01.809489 13805 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:02:01.674474 18427 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:01.734313 14622 rpcz_store.cc:276] 0807 08:02:00.529332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.529472 (+ 140us) service_pool.cc:227] Handling call
0807 08:02:00.562524 (+ 33052us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:00.569596 (+ 7072us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:00.569640 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:00.581763 (+ 12123us) log.cc:842] Serialized 26 byte log entry
0807 08:02:01.448927 (+867164us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:02:01.448979 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.449069 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:01.681079 (+232010us) raft_consensus.cc:1671] finished
0807 08:02:01.681098 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.681207 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":145,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":4785,"thread_start_us":817435,"threads_started":1,"wal-append.queue_time_us":1076029}
I20260807 08:02:01.726284 17943 raft_consensus.cc:2726] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:01.744355 18381 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.674670 18410 consensus_queue.cc:1046] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.061s
I20260807 08:02:01.823190 14580 leader_election.cc:302] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:01.726305 18577 raft_consensus.cc:729] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:01.763337 18308 raft_consensus.cc:3032] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:02:01.724860 18482 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.769109 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:01.772687 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:02:01.774354 13579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56594 (request call id 1849) took 1789 ms (client timeout 1684 ms). Trace:
I20260807 08:02:01.776552 18076 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:02:01.778117 18303 raft_consensus.cc:3037] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:01.778716 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:01.779178 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:01.789230 14583 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:01.697431 18512 consensus_queue.cc:260] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.736344 18447 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:01.804458 13591 raft_consensus.cc:603] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:02:01.804767 14630 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
W20260807 08:02:01.813299 13588 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s user 0.006s sys 0.039s
W20260807 08:02:01.798249 13700 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.025s sys 0.037s
W20260807 08:02:01.816447 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 365537us
I20260807 08:02:01.808483 13702 rpcz_store.cc:276] 0807 08:02:00.074596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.146399 (+ 71803us) service_pool.cc:227] Handling call
0807 08:02:00.385814 (+239415us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:02:00.448690 (+ 62876us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.730483 (+1281793us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400081bd0. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:01.730686 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":549773184}
I20260807 08:02:01.741752 18201 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:01.806263 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 103045us
I20260807 08:02:01.729907 18521 raft_consensus.cc:686] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:01.752640 13703 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:02:01.824740 13708 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
W20260807 08:02:01.826252 13710 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.020s sys 0.011s
I20260807 08:02:01.833720 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:02:01.966534 13710 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s user 0.020s sys 0.011s
I20260807 08:02:01.834698 18090 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:02:01.839332 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 386968us
I20260807 08:02:01.840158 13817 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:02:01.840410 13579 rpcz_store.cc:269] 0807 08:01:59.984921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.077826 (+ 92905us) service_pool.cc:227] Handling call
0807 08:02:01.774208 (+1696382us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54001e3a54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:01.774294 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1141707648}
I20260807 08:02:01.840060 13811 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 9.
I20260807 08:02:01.805033 18419 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:01.979997 13703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1857) took 1102 ms. Trace:
I20260807 08:02:01.883291 13590 raft_consensus.cc:1226] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20260807 08:02:01.803653 18344 raft_consensus.cc:2781] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:01.739769 18443 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:01.907457 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
W20260807 08:02:01.824936 18478 consensus_meta.cc:287] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.746s user 0.013s sys 0.002s
I20260807 08:02:01.816478 18253 raft_consensus.cc:1067] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Signalling peer 7a889001d6494fe5b7cbbe7da1dae302 to start an election
W20260807 08:02:01.824788 18370 consensus_meta.cc:287] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.799s user 0.002s sys 0.011s
I20260807 08:02:01.822847 18445 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:02:01.838061 18308 raft_consensus.cc:729] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:01.791467 18496 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.846352 18461 raft_consensus.cc:2781] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:01.925941 13807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54086 (request call id 2106) took 1053 ms. Trace:
W20260807 08:02:01.884620 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 59689.6us
I20260807 08:02:01.844149 18345 raft_consensus.cc:2781] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260807 08:02:01.927212 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 138069us
I20260807 08:02:01.929338 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:01.881814 18425 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:01.735014 18378 consensus_queue.cc:260] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.822563 18489 raft_consensus.cc:504] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:01.983992 18344 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:02:01.972888 13700 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s user 0.026s sys 0.046s
I20260807 08:02:01.974174 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.006917 13807 rpcz_store.cc:276] 0807 08:02:00.872060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.872252 (+ 192us) service_pool.cc:227] Handling call
0807 08:02:00.872514 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:00.872713 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 4
0807 08:02:01.649713 (+777000us) log.cc:842] Serialized 14 byte log entry
0807 08:02:01.925621 (+275908us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:02:01.925668 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0807 08:02:01.925715 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.925773 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:01.925889 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":204,"prepare.queue_time_us":906898,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":48,"raft.queue_time_us":1900,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":51892,"thread_start_us":1052080,"threads_started":2}
I20260807 08:02:01.978435 13540 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:01.979548 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:01.980495 13703 rpcz_store.cc:276] 0807 08:02:00.888189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.905821 (+ 17632us) service_pool.cc:227] Handling call
0807 08:02:01.979849 (+1074028us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x7b5400012cd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:01.979944 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1197160192}
I20260807 08:02:01.986963 18590 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:01.930825 18423 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:01.926961 18539 raft_consensus.cc:3032] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 LEADER]: Stepping down as leader of term 8
I20260807 08:02:01.925372 17654 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:01.996729 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:01.987504 13769 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:01.985385 18419 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.007550 13589 raft_consensus.cc:3037] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:02.008965 14583 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:02.006887 18461 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:01.984520 18090 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:02.026880 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 48542.5us
I20260807 08:02:02.009994 18345 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:01.983922 18514 consensus_queue.cc:1046] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260807 08:02:02.043598 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.044378 13585 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 08:02:02.027887 13812 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:02:01.995170 18445 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:02.019615 18344 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.048020 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.049713 13540 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:02.003295 18370 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:01.837710 18380 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.031332 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:02.034101 13708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1864) took 1049 ms. Trace:
I20260807 08:02:01.981580 18447 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:02.063030 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 53561.2us
I20260807 08:02:02.056512 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:02.052821 13708 rpcz_store.cc:276] 0807 08:02:00.984867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:01.135576 (+150709us) service_pool.cc:227] Handling call
0807 08:02:02.033909 (+898333us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b5400180954. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:02.034033 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1317452672}
I20260807 08:02:02.061905 13537 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:02.058061 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.028302 18376 raft_consensus.cc:2726] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:02.082891 18376 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:02.083176 18380 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.083345 18376 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:02.084122 18594 log.cc:924] Time spent T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260807 08:02:02.063621 18448 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260807 08:02:02.049091 18345 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.068166 13705 raft_consensus.cc:1226] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260807 08:02:02.070245 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.025462 18539 raft_consensus.cc:729] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:02.071269 13701 raft_consensus.cc:1226] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:02.075470 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:02.012713 18196 consensus_queue.cc:1046] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
W20260807 08:02:02.077992 18602 log.cc:924] Time spent T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
I20260807 08:02:01.965593 18577 consensus_queue.cc:260] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.095347 13587 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 10.
W20260807 08:02:02.079948 18597 log.cc:924] Time spent T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.060s user 0.001s sys 0.001s
I20260807 08:02:02.081174 14619 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 10 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 9. Current term is 10. Ops: []
I20260807 08:02:02.081422 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.095012 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:02.069206 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:02.125101 14627 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:02.076074 13811 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c"
dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:59332
I20260807 08:02:02.127652 13811 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:02:02.128108 13811 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:02:02.098562 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.018574 18425 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:02.137432 18603 log.cc:924] Time spent T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20260807 08:02:02.139183 18425 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.140390 18577 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.144743 18344 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.125578 18413 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.146991 18447 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.147922 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.148907 13708 raft_consensus.cc:2379] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:02:02.150113 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.137326 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.150727 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 500) took 1118 ms. Trace:
I20260807 08:02:02.062700 18423 raft_consensus.cc:2781] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 08:02:02.108481 13589 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 9.
I20260807 08:02:02.074230 13706 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:02:02.037642 18592 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 08:02:02.119442 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.040330 18461 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.051517 18445 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:02.135143 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:02.136372 18090 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.136471 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:02.136812 18381 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.067618 18268 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:02.137276 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 505) took 1078 ms. Trace:
I20260807 08:02:02.151166 13702 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:02:02.103873 13579 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:02:02.154860 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.085358 18376 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.155920 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.151049 13710 rpcz_store.cc:276] 0807 08:02:01.032794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:01.283908 (+251114us) service_pool.cc:227] Handling call
0807 08:02:01.284229 (+ 321us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:02:01.284599 (+ 370us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:01.284628 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:02:01.285127 (+ 499us) log.cc:842] Serialized 52 byte log entry
0807 08:02:01.966935 (+681808us) raft_consensus.cc:1631] Marking committed up to 8
0807 08:02:02.002580 (+ 35645us) log.cc:842] Serialized 14 byte log entry
0807 08:02:02.024418 (+ 21838us) log.cc:842] Serialized 14 byte log entry
0807 08:02:02.039586 (+ 15168us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:02.039666 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:02.150526 (+110860us) raft_consensus.cc:1671] finished
0807 08:02:02.150545 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:02.150673 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83302,"prepare.queue_time_us":75545,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":25412,"raft.queue_time_us":37457,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":19895,"spinlock_wait_cycles":14247040,"thread_start_us":681474,"threads_started":1,"wal-append.queue_time_us":694453}
I20260807 08:02:02.158232 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.225488 13706 raft_consensus.cc:3037] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 9
I20260807 08:02:02.223445 13711 rpcz_store.cc:276] 0807 08:02:01.058540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:01.489081 (+430541us) service_pool.cc:227] Handling call
0807 08:02:01.489444 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:01.491779 (+ 2335us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:01.520978 (+ 29199us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:01.521458 (+ 480us) log.cc:842] Serialized 26 byte log entry
0807 08:02:01.930304 (+408846us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:02:01.987232 (+ 56928us) log.cc:842] Serialized 14 byte log entry
0807 08:02:01.995377 (+ 8145us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.995443 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:02.137059 (+141616us) raft_consensus.cc:1671] finished
0807 08:02:02.137089 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:02.137224 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43382,"prepare.queue_time_us":75093,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":51120,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":11530112,"thread_start_us":364225,"threads_started":1,"wal-append.queue_time_us":497469}
I20260807 08:02:02.131084 18482 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:02.203182 18612 log.cc:924] Time spent T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260807 08:02:02.099606 13588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33050 (request call id 676) took 1253 ms. Trace:
I20260807 08:02:02.162248 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.170645 18544 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.178444 14627 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:02:02.234673 13708 raft_consensus.cc:2396] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:02:02.235488 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:02.193177 14626 raft_consensus.cc:1226] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260807 08:02:02.194299 14622 raft_consensus.cc:1226] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [term 8 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:02.196863 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.194535 18592 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:02.237069 13814 raft_consensus.cc:2379] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 4: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
W20260807 08:02:02.129143 18403 consensus_meta.cc:287] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.548s user 0.007s sys 0.003s
I20260807 08:02:02.199641 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.048362 18308 consensus_queue.cc:260] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.234210 13588 rpcz_store.cc:276] 0807 08:02:00.845808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:01.115014 (+269206us) service_pool.cc:227] Handling call
0807 08:02:01.115432 (+ 418us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:01.115832 (+ 400us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:01.115875 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:01.127598 (+ 11723us) log.cc:842] Serialized 26 byte log entry
0807 08:02:01.974381 (+846783us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:02:01.974434 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.974524 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:02.099426 (+124902us) raft_consensus.cc:1671] finished
0807 08:02:02.099444 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:02.099557 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":106278,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":3773,"thread_start_us":768376,"threads_started":1,"wal-append.queue_time_us":747592}
W20260807 08:02:02.213688 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 180663us
I20260807 08:02:02.215085 14619 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:02:02.123049 18448 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:02.245797 14626 raft_consensus.cc:1261] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 08:02:02.223358 18268 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:02:02.159554 18542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.000s sys 0.017s
I20260807 08:02:02.160471 13661 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:02:02.229375 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 130035us
I20260807 08:02:02.224968 13808 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:02:02.024927 18590 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.232348 13706 raft_consensus.cc:2454] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:02:02.235987 14627 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:02:02.240675 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.257473 13579 raft_consensus.cc:3037] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:02:02.243952 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.205819 14620 raft_consensus.cc:1226] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
W20260807 08:02:02.244057 13769 leader_election.cc:334] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.694s (SENT)
I20260807 08:02:02.231787 18607 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:02.050340 18521 consensus_queue.cc:237] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.258621 13588 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 9.
I20260807 08:02:02.249507 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.253767 13700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1861) took 1337 ms. Trace:
I20260807 08:02:02.254840 14580 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:02.259284 13769 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.265292 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.265980 13588 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:02.265179 13700 rpcz_store.cc:276] 0807 08:02:00.916088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:01.278367 (+362279us) service_pool.cc:227] Handling call
0807 08:02:01.278792 (+ 425us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:02:01.279246 (+ 454us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:01.279290 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:02:01.280044 (+ 754us) log.cc:842] Serialized 52 byte log entry
0807 08:02:02.015396 (+735352us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:02:02.025290 (+ 9894us) log.cc:842] Serialized 14 byte log entry
0807 08:02:02.098784 (+ 73494us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:02.098851 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:02.253535 (+154684us) raft_consensus.cc:1671] finished
0807 08:02:02.253555 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:02.253695 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77745,"prepare.queue_time_us":11654,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":5543,"raft.queue_time_us":98222,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":22873,"thread_start_us":734811,"threads_started":1,"wal-append.queue_time_us":641956}
I20260807 08:02:02.266829 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.261643 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:02.262953 18268 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:02:02.264206 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 103843us
I20260807 08:02:02.241875 18482 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.276641 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:02:02.277137 13769 leader_election.cc:334] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.807s (SENT)
I20260807 08:02:02.277344 13590 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 9: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:02.271934 13657 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.277580 13769 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:02.278239 13703 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:02:02.280848 13579 raft_consensus.cc:2454] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
W20260807 08:02:02.254334 18542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s user 0.000s sys 0.018s
I20260807 08:02:02.240562 18592 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:02.252764 18308 raft_consensus.cc:3037] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.285485 13657 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:02.284022 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.286548 13588 raft_consensus.cc:2379] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 9: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:02.281503 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:02:02.287845 13769 leader_election.cc:334] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 1.855s (SENT)
W20260807 08:02:02.274190 18493 consensus_meta.cc:287] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.525s user 0.010s sys 0.000s
I20260807 08:02:02.288374 13769 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:02.286999 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.233884 18345 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:02.306116 18625 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260807 08:02:02.307171 18345 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.307368 18590 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.307824 14630 raft_consensus.cc:2354] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8.
I20260807 08:02:02.287972 17915 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.275177 18370 raft_consensus.cc:2781] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election won for term 9
I20260807 08:02:02.308153 18590 consensus_queue.cc:1046] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260807 08:02:02.287004 13579 raft_consensus.cc:2362] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7. Re-sending same reply.
I20260807 08:02:02.309096 17681 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 08:02:02.292405 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:02.316783 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.309679 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.318681 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.255771 18443 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.285976 18482 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.295459 13661 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:02.296268 18402 consensus_queue.cc:1035] T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:02:02.298272 13540 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.293417 18447 raft_consensus.cc:2726] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:02.309695 18496 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.300411 13585 raft_consensus.cc:2396] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:02:02.331679 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:02.332649 13812 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 9: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:02:02.249025 18448 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.333278 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.303893 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.308629 18370 raft_consensus.cc:686] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:02.308389 17915 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.292342 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.339635 13807 raft_consensus.cc:2379] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:02:02.314592 13805 raft_consensus.cc:2379] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 10: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:02:02.294973 13590 raft_consensus.cc:3037] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:02.290628 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.341038 13588 raft_consensus.cc:2379] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 6: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:02.341909 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.342015 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.343050 13592 raft_consensus.cc:2354] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9.
I20260807 08:02:02.343360 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.309480 18590 raft_consensus.cc:3032] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:02:02.281978 18591 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.329304 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.329344 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:02:02.288133 18515 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:02.328161 18076 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260807 08:02:02.314043 18617 log.cc:924] Time spent T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
I20260807 08:02:02.359381 13584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56592 (request call id 1971) took 2198 ms. Trace:
I20260807 08:02:02.359766 13584 rpcz_store.cc:276] 0807 08:02:00.160721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.816152 (+655431us) service_pool.cc:227] Handling call
0807 08:02:00.844411 (+ 28259us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:01.607418 (+763007us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:01.607460 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:01.672917 (+ 65457us) log.cc:842] Serialized 26 byte log entry
0807 08:02:01.992802 (+319885us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:02:01.992833 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:01.992898 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:02.359173 (+366275us) raft_consensus.cc:1671] finished
0807 08:02:02.359192 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:02.359312 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":363875,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":4706,"thread_start_us":1025120,"threads_started":2,"wal-append.queue_time_us":396158}
I20260807 08:02:02.289228 18489 leader_election.cc:288] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 10 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:02.287855 18615 log.cc:924] Time spent T 74ccb94230614cc48e846580ad82e994 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
W20260807 08:02:02.347129 18628 log.cc:924] Time spent T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260807 08:02:02.347644 13661 leader_election.cc:398] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9.
I20260807 08:02:02.380079 13661 leader_election.cc:401] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
W20260807 08:02:02.350169 18632 log.cc:924] Time spent T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20260807 08:02:02.364962 18601 log.cc:924] Time spent T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260807 08:02:02.351833 18590 raft_consensus.cc:729] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:02.284958 18600 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:02.365010 18376 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:02.353065 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.330415 18542 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:02.357981 18076 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:02.361491 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.362968 13540 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.311179 18419 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.371428 13811 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.384162 13705 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:02:02.379160 13590 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 10.
I20260807 08:02:02.408284 13590 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260807 08:02:02.309497 17681 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:02.409499 13590 raft_consensus.cc:1261] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 08:02:02.409777 17681 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Advancing to term 11
I20260807 08:02:02.410503 13540 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.352607 13579 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:02.325948 18493 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.344074 13588 raft_consensus.cc:2362] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 3. Re-sending same reply.
I20260807 08:02:02.385774 18344 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:02.424062 13588 raft_consensus.cc:1226] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:02.364560 17915 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.364804 18544 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.405272 13584 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:02.405634 13809 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:02:02.409155 18640 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.392673 18381 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) with higher term. Message: Invalid argument: T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9.
I20260807 08:02:02.435877 13709 raft_consensus.cc:1226] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:02.362592 18604 consensus_queue.cc:1035] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260807 08:02:02.426888 13579 raft_consensus.cc:1261] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260807 08:02:02.422334 13811 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.427798 13588 raft_consensus.cc:1226] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9. Ops: [6.7-8.8]
W20260807 08:02:02.429095 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 100134us
I20260807 08:02:02.400256 17654 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.379551 18515 consensus_queue.cc:1035] T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20260807 08:02:02.420867 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.404088 18447 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.401737 18376 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:02.368811 18591 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.406826 18076 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:02:02.498333 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:02:02.490303 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 80788.4us
I20260807 08:02:02.422739 18428 raft_consensus.cc:2726] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:02:02.419176 18605 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:02:02.363432 18370 consensus_queue.cc:237] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.418020 18634 raft_consensus.cc:2741] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260807 08:02:02.486439 13592 raft_consensus.cc:2396] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260807 08:02:02.499536 13540 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.489996 18381 consensus_queue.cc:1046] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260807 08:02:02.516010 13540 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:02:02.516265 13769 leader_election.cc:334] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 2.082s (SENT)
I20260807 08:02:02.516767 13769 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
W20260807 08:02:02.507825 18622 log.cc:924] Time spent T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20260807 08:02:02.516193 18381 raft_consensus.cc:3032] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:02:02.519287 18521 raft_consensus.cc:2726] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:02.519454 18381 raft_consensus.cc:729] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:02:02.520035 18644 log.cc:924] Time spent T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
W20260807 08:02:02.520517 13769 leader_election.cc:334] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037): Timed out: RequestConsensusVote RPC to 127.11.140.65:41037 timed out after 2.090s (SENT)
I20260807 08:02:02.509150 18268 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.521051 13769 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:02.521895 18521 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.522804 18268 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.512840 18461 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.524986 18461 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.525942 14618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 649) took 1686 ms. Trace:
I20260807 08:02:02.526201 18646 raft_consensus.cc:2781] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election won for term 9
I20260807 08:02:02.526683 18646 raft_consensus.cc:686] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [term 9 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:02.526342 14618 rpcz_store.cc:276] 0807 08:02:00.839695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.839847 (+ 152us) service_pool.cc:227] Handling call
0807 08:02:00.856072 (+ 16225us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:02:01.610593 (+754521us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:01.610629 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:02:01.672953 (+ 62324us) log.cc:842] Serialized 52 byte log entry
0807 08:02:01.993493 (+320540us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:02:02.019314 (+ 25821us) log.cc:842] Serialized 14 byte log entry
0807 08:02:02.021970 (+ 2656us) log.cc:842] Serialized 14 byte log entry
0807 08:02:02.037755 (+ 15785us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:02.037833 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:02.081699 (+ 43866us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b18002dfd00. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043764 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:02:02.525724 (+444025us) raft_consensus.cc:1671] finished
0807 08:02:02.525747 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:02.525879 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41757,"prepare.queue_time_us":78594,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":3469,"raft.queue_time_us":114089,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":39527,"spinlock_wait_cycles":44811776,"thread_start_us":980645,"threads_started":2,"wal-append.queue_time_us":623322}
I20260807 08:02:02.527655 18646 consensus_queue.cc:237] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.498406 18376 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.529237 13769 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:02:02.529498 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 110864us
I20260807 08:02:02.532876 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:02.534192 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.534703 13769 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.534953 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.536912 13706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.501879 18600 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:02.533496 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.537317 13700 raft_consensus.cc:1261] T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 08:02:02.520470 18381 consensus_queue.cc:260] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.538144 13540 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.529464 18428 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:02:02.546000 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.498347 18447 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.548086 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.533435 18436 raft_consensus.cc:2741] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260807 08:02:02.518587 18605 raft_consensus.cc:686] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:02.531703 18482 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.553354 13540 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.551112 13769 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.538784 18591 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.555066 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.484037 18640 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:02.552270 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 115358us
I20260807 08:02:02.540428 18600 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:02.555259 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.539770 18577 raft_consensus.cc:2726] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:02:02.555732 13711 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:02:02.556593 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.541127 18493 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.500077 18590 consensus_queue.cc:260] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.559978 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.561504 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.568714 13705 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:02:02.562494 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.564162 13711 raft_consensus.cc:1226] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:02.568895 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.565771 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.569620 13588 raft_consensus.cc:2362] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 9. Re-sending same reply.
I20260807 08:02:02.566339 18515 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:02.547206 18448 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.553942 18544 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:02.570384 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.572407 18591 consensus_queue.cc:1046] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:02:02.573009 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.573676 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.542403 18268 consensus_queue.cc:1046] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260807 08:02:02.587558 18605 consensus_queue.cc:237] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.589852 18419 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.578002 17681 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.591660 13700 raft_consensus.cc:2379] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 11: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:02.575356 13706 raft_consensus.cc:2396] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260807 08:02:02.573259 18640 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.592607 17681 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 11 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.568202 18600 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.574539 13710 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260807 08:02:02.518896 17654 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.577898 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
W20260807 08:02:02.575357 18616 log.cc:924] Time spent T 00000000000000000000000000000000 P 5d5229fe29be45b28bd5a2e9d3d998b1: Append to log took a long time: real 0.207s user 0.000s sys 0.002s
I20260807 08:02:02.560148 18428 raft_consensus.cc:686] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [term 6 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:02.567754 18482 raft_consensus.cc:2726] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260807 08:02:02.574250 13769 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:02.558251 18381 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.591522 14583 leader_election.cc:302] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.597843 13590 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.576416 18515 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:02.581357 13592 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:02:02.576469 18436 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:02.606393 13769 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:02.568879 13540 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.609664 13588 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:02:02.615841 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.616444 13584 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 9.
I20260807 08:02:02.616611 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.620312 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:02:02.625798 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 57031.8us
I20260807 08:02:02.627691 13540 leader_election.cc:302] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.630275 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:02.609094 18590 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:02:02.636497 13508 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 7a889001d6494fe5b7cbbe7da1dae302 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } }
I20260807 08:02:02.637727 18419 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 08:02:02.612788 18515 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.646340 13590 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:02:02.645287 18419 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:02.648092 18655 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.648283 18419 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:02.608991 18308 consensus_queue.cc:1035] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.037s
I20260807 08:02:02.648748 13661 leader_election.cc:302] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:02.601189 18445 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.630539 18482 raft_consensus.cc:2781] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:02:02.633033 18268 raft_consensus.cc:2726] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:02.631397 17681 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:02.648607 18655 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.678076 17681 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:02.678599 17681 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.678856 18445 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.679626 18655 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.681747 18647 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:02.682462 18647 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.683086 18600 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.685135 18647 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.685708 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.638803 18577 consensus_queue.cc:1046] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260807 08:02:02.653906 18592 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.690093 13817 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 10.
I20260807 08:02:02.669092 18482 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:02.690856 18482 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:02:02.691119 18592 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:02.658396 18645 raft_consensus.cc:2781] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:02:02.692281 18645 raft_consensus.cc:686] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:02.652918 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:02:02.694590 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 88295.5us
I20260807 08:02:02.695271 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.695485 13769 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:02.664566 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.696985 13711 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:02.670784 13588 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:02:02.677641 18521 raft_consensus.cc:2726] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:02.677824 13808 raft_consensus.cc:1226] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6. Ops: []
I20260807 08:02:02.702157 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.701298 13769 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters: 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.702891 13711 raft_consensus.cc:2379] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 7: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:02.703372 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.703992 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.704074 13592 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:02:02.704241 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.638511 18428 consensus_queue.cc:237] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.648854 18656 raft_consensus.cc:2726] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:02:02.693382 18645 consensus_queue.cc:237] T fb6a66ac7784477f904283ba3204bac6 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.705814 13711 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260807 08:02:02.706175 13811 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.706712 13710 raft_consensus.cc:2379] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 10: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:02.707420 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.707528 13661 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:02.708722 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.639029 18448 consensus_queue.cc:1046] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260807 08:02:02.712386 13805 raft_consensus.cc:2379] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:02:02.713038 13769 leader_election.cc:302] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.715991 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:02.703220 18445 consensus_queue.cc:1035] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260807 08:02:02.729116 17654 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.721401 18591 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:02.729794 18591 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:02.730204 18591 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.730880 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1866) took 1124 ms. Trace:
I20260807 08:02:02.731254 13709 rpcz_store.cc:276] 0807 08:02:01.606034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:01.606154 (+ 120us) service_pool.cc:227] Handling call
0807 08:02:01.670174 (+ 64020us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:02.436316 (+766142us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:02.730656 (+294340us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54001d38d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:02.730824 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1760548736}
I20260807 08:02:02.733706 18447 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.734326 18445 raft_consensus.cc:2726] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 08:02:02.735881 13580 raft_consensus.cc:1261] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 08:02:02.736210 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.736800 13584 raft_consensus.cc:2362] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4. Re-sending same reply.
I20260807 08:02:02.671526 18376 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.738947 18376 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.739454 18376 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:02.739527 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.739784 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 503) took 1687 ms. Trace:
I20260807 08:02:02.739967 18376 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.740309 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.711098 18521 raft_consensus.cc:2781] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:02.741439 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.741755 18521 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:02.741994 13592 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.747428 13580 raft_consensus.cc:2454] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:02.748222 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.755369 18482 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:02.724918 18631 log.cc:924] Time spent T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.395s user 0.001s sys 0.000s
I20260807 08:02:02.725270 18515 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.756742 13592 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:02:02.724144 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.720330 13657 leader_election.cc:302] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.761849 18482 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.724700 18662 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.763556 18662 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.764276 13657 leader_election.cc:302] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.765621 18658 raft_consensus.cc:2726] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:02.766204 18658 raft_consensus.cc:2726] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260807 08:02:02.766590 13657 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.773119 18402 raft_consensus.cc:2726] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 08:02:02.742642 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.742166 18521 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.740175 13701 rpcz_store.cc:276] 0807 08:02:01.052719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:01.407972 (+355253us) service_pool.cc:227] Handling call
0807 08:02:01.411476 (+ 3504us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:02.095690 (+684214us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:02.739574 (+643884us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400082fd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:02.739730 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1516903168}
I20260807 08:02:02.743548 13540 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:02.756282 18436 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 08:02:02.719329 18642 raft_consensus.cc:2726] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260807 08:02:02.757862 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.783291 13814 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:02:02.758097 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:02.784013 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.784611 13701 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:02:02.760737 18591 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.786814 18591 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.788779 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.771790 13769 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.742763 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.703334 18592 raft_consensus.cc:2781] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 08:02:02.780478 13811 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:02:02.790371 13584 raft_consensus.cc:2379] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 9: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:02.790617 13592 raft_consensus.cc:1226] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260807 08:02:02.792483 13701 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:02:02.759232 18515 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:02.793826 17654 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:02.794107 18662 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.792286 13816 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 9.
I20260807 08:02:02.795823 13769 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:02.818854 13540 leader_election.cc:302] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:02.820731 13769 leader_election.cc:302] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.827742 13708 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:02:02.794400 17654 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.845424 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:02.796473 18600 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.828436 18591 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:02:02.843174 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 50627.8us
I20260807 08:02:02.844051 18378 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:02.793066 18482 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:02.845013 18515 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:02.851054 18436 raft_consensus.cc:2726] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 08:02:02.842029 18592 raft_consensus.cc:686] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:02.798971 18642 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:02.882279 18592 consensus_queue.cc:237] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.850086 18076 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.886901 17654 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.887929 18076 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.843196 18590 raft_consensus.cc:2781] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader election won for term 5
I20260807 08:02:02.889147 18590 raft_consensus.cc:686] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [term 5 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:02.890368 18590 consensus_queue.cc:237] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.854487 18542 raft_consensus.cc:2726] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 08:02:02.893136 18076 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.893625 18076 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.856573 18445 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.038s
I20260807 08:02:02.895506 18076 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.864699 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.898437 13584 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.849917 18376 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.867655 18515 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:02.902160 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.904486 13701 raft_consensus.cc:2362] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4. Re-sending same reply.
I20260807 08:02:02.907605 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.908754 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.908943 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.902518 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.909793 13592 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 9.
I20260807 08:02:02.910727 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.908417 13814 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 5.
I20260807 08:02:02.846002 17681 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.920913 13591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:39822 (request call id 20) took 2105 ms. Trace:
I20260807 08:02:02.921270 13591 rpcz_store.cc:276] 0807 08:02:00.806964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:00.831391 (+ 24427us) service_pool.cc:227] Handling call
0807 08:02:02.912659 (+2081268us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b5400204ed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf65bc3 00007f6b21eaf81a 00007f6b1b5a2d04 00007f6b1b5a2c92 00007f6b1b5a2bc1 00007f6b1b5a2b3c 00007f6b1b5a1df2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:02.912772 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2038167424}
I20260807 08:02:02.869078 18447 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:02.911720 18515 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.922998 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.926435 13769 leader_election.cc:302] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:02.867075 18482 raft_consensus.cc:482] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260807 08:02:02.914168 18661 log.cc:924] Time spent T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20260807 08:02:02.925652 13705 raft_consensus.cc:1226] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:02.925438 18436 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260807 08:02:02.931470 18577 consensus_queue.cc:1046] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 08:02:02.931680 18600 consensus_queue.cc:1035] T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:02:02.933619 18445 raft_consensus.cc:2781] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:02:02.934132 18445 raft_consensus.cc:686] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:02.935014 18445 consensus_queue.cc:237] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.865911 18591 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:02.937736 18591 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.927148 18447 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.939026 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:02.940038 13580 raft_consensus.cc:2379] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:02.943552 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:02.944252 13580 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.947496 18581 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:02.948170 18581 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.957461 18581 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.958632 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:02.959580 13705 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:02:02.960034 18419 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:02.961057 13540 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:02.957743 17681 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:02.962182 18419 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.929472 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.967787 13701 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:02.972074 13584 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:02.978914 13701 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:02:02.987638 13580 raft_consensus.cc:2454] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:02:02.988242 18584 raft_consensus.cc:3032] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:02:02.992887 18584 raft_consensus.cc:729] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:02.989552 13591 raft_consensus.cc:1226] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:02.989413 13769 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:02.958508 18447 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:02.995205 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:02.930673 18482 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:02.997409 13769 leader_election.cc:302] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:02.997200 13657 leader_election.cc:302] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:02.998188 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:02.999189 13804 raft_consensus.cc:2379] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 5: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:02:02.998335 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:02:02.999902 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 47476.7us
I20260807 08:02:03.001840 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:03.004235 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.981616 18581 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 08:02:03.005434 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:03.005580 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:03.006050 13812 raft_consensus.cc:3037] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 7
I20260807 08:02:03.008280 18681 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 08:02:03.008745 18512 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:03.009351 17943 raft_consensus.cc:2781] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:03.009295 13657 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:03.009917 17943 raft_consensus.cc:482] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:03.010319 17943 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:03.011096 13657 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:02.987272 18436 consensus_queue.cc:1046] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260807 08:02:03.014086 13703 raft_consensus.cc:1261] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 08:02:03.006841 18398 raft_consensus.cc:2781] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:02:03.014569 18427 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260807 08:02:03.014235 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:03.014842 18398 raft_consensus.cc:686] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:03.015287 13588 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:03.015336 18427 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:03.015748 18398 consensus_queue.cc:237] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.021871 13769 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:03.017089 13661 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:03.027938 13588 raft_consensus.cc:2454] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 10.
I20260807 08:02:03.029955 13661 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:03.030675 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:02.993754 18584 consensus_queue.cc:260] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.032699 13701 raft_consensus.cc:1261] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 08:02:03.038259 13661 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:03.008747 18681 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:03.039510 17681 raft_consensus.cc:2781] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Leader election won for term 10
I20260807 08:02:03.040078 17681 raft_consensus.cc:686] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:03.040974 17681 consensus_queue.cc:237] T 069b769373e144929c9253bf3fdd0ab4 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.041874 18584 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:03.043080 18436 raft_consensus.cc:2726] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260807 08:02:03.036233 18478 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:02:03.044210 18478 raft_consensus.cc:686] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:03.037284 13769 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:03.039395 18664 consensus_queue.cc:1046] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260807 08:02:03.027144 18076 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:03.022758 18419 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.051817 18436 consensus_queue.cc:1035] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:02:03.046427 18591 raft_consensus.cc:2726] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260807 08:02:03.055393 13769 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:03.049372 18542 raft_consensus.cc:2781] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:02:03.048066 18681 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:03.057569 18656 consensus_queue.cc:1035] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.005s
I20260807 08:02:03.056497 13588 raft_consensus.cc:1261] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 08:02:03.070600 18445 raft_consensus.cc:2781] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader election won for term 4
I20260807 08:02:03.069780 18376 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.075552 18376 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.053417 18076 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.054786 18419 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.084172 18515 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.071349 18478 consensus_queue.cc:237] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.086575 18515 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:03.082118 18662 consensus_queue.cc:1035] T 76bbc935aada4ceab2774e363186745d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260807 08:02:03.087487 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:03.069975 18542 raft_consensus.cc:686] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:03.089250 18542 consensus_queue.cc:237] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.091051 18076 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.092039 18521 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:03.093693 18688 log.cc:924] Time spent T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260807 08:02:03.094064 18521 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:03.089267 18419 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.094640 13807 raft_consensus.cc:1226] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:03.094925 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:03.099406 13588 raft_consensus.cc:1226] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20260807 08:02:03.101837 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:03.102994 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:03.103296 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:03.103771 13769 leader_election.cc:302] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:03.116945 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:03.117684 13709 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:03.119721 13804 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 9.
I20260807 08:02:03.134893 13703 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260807 08:02:03.141558 13584 raft_consensus.cc:1261] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260807 08:02:03.153793 18634 consensus_queue.cc:1046] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260807 08:02:03.156744 18687 log.cc:924] Time spent T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.168s user 0.002s sys 0.000s
I20260807 08:02:03.125360 18521 raft_consensus.cc:2726] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260807 08:02:03.158879 13709 raft_consensus.cc:2454] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:02:03.135053 18581 raft_consensus.cc:482] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:03.122844 18419 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.172283 13817 raft_consensus.cc:1226] T 6962d9b636574c5e80f4e57a227e52e2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:03.174019 13769 leader_election.cc:302] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:03.175256 18398 raft_consensus.cc:2781] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:02:03.175612 18445 raft_consensus.cc:686] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:03.175834 18398 raft_consensus.cc:686] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:03.140749 17915 consensus_queue.cc:1046] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260807 08:02:03.174746 18655 consensus_queue.cc:1035] T 1588644cd4f943a7b5d776326af48218 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260807 08:02:03.170799 18581 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:03.205937 13812 raft_consensus.cc:2454] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 7.
I20260807 08:02:03.207347 13657 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:03.172853 18478 consensus_queue.cc:1046] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260807 08:02:03.177053 18445 consensus_queue.cc:237] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.222668 18681 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.233870 13588 raft_consensus.cc:1226] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260807 08:02:03.235250 18681 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.235833 18681 consensus_queue.cc:1046] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 08:02:03.211189 18398 consensus_queue.cc:237] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.239786 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:03.240640 13588 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:03.181087 18419 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.275382 18514 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:03.276250 18419 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.275939 18514 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.186455 17943 raft_consensus.cc:504] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.278438 18514 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.271327 13588 raft_consensus.cc:2454] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 10.
I20260807 08:02:03.249351 18542 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.281105 18542 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.274170 18376 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.197077 18482 raft_consensus.cc:504] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.254401 13584 raft_consensus.cc:1261] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 08:02:03.279862 13769 leader_election.cc:302] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:03.289915 13592 raft_consensus.cc:2379] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 7: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:03.293717 18581 raft_consensus.cc:504] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.282545 18634 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:03.295528 14629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 651) took 1624 ms. Trace:
I20260807 08:02:03.295275 18634 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.298970 18482 leader_election.cc:288] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:03.289150 18664 raft_consensus.cc:2781] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Leader election won for term 10
I20260807 08:02:03.300062 18664 raft_consensus.cc:686] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:03.301132 18664 consensus_queue.cc:237] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.304109 18542 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.305899 18634 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.306911 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:03.307670 13584 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
W20260807 08:02:03.310160 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 1.05532e+06us
I20260807 08:02:03.311897 13769 leader_election.cc:398] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8.
I20260807 08:02:03.312167 13769 leader_election.cc:401] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260807 08:02:03.312299 13508 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2de183214b52498faef7f8a8f74a2adc, TS 77da774f2bb54cf28302d53d0db224ad -> TS 22ff0a1f4a4241d794ab182432c85893 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:03.312639 13508 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2de183214b52498faef7f8a8f74a2adc, TS 77da774f2bb54cf28302d53d0db224ad -> TS 22ff0a1f4a4241d794ab182432c85893 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.314324 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.315006 14630 raft_consensus.cc:1261] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260807 08:02:03.315490 18345 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:02:03.315899 18345 raft_consensus.cc:482] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:03.316210 18345 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:03.316237 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.316996 14618 raft_consensus.cc:3032] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Stepping down as leader of term 3
I20260807 08:02:03.317341 14618 raft_consensus.cc:729] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:03.317704 18489 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:03.318221 18489 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.318292 14618 consensus_queue.cc:260] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:03.319360 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 570980us
I20260807 08:02:03.319676 14618 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:02:03.320492 18489 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:03.324496 18592 raft_consensus.cc:3032] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:02:03.324956 18592 raft_consensus.cc:729] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
W20260807 08:02:03.326027 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 734530us
I20260807 08:02:03.325222 14625 raft_consensus.cc:2932] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
I20260807 08:02:03.328262 18432 raft_consensus.cc:2726] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 08:02:03.329178 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.326725 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:03.329854 14623 raft_consensus.cc:3037] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:03.331652 14583 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:03.332329 18584 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:03.332901 18584 raft_consensus.cc:2726] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260807 08:02:03.324615 18398 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:03.337867 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:03.350366 13812 raft_consensus.cc:2454] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:02:03.352586 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:03.355577 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:03.339306 13537 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:03.339697 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:03.360077 13811 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 4.
I20260807 08:02:03.340093 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.361721 14630 raft_consensus.cc:2379] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:02:03.362181 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:03.363124 13814 raft_consensus.cc:2379] T e67bd63471c94c19a092c8f1aef64408 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 7: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:03.342880 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.335194 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:03.295913 14629 rpcz_store.cc:276] 0807 08:02:01.671369 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:01.671524 (+ 155us) service_pool.cc:227] Handling call
0807 08:02:01.702033 (+ 30509us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:02:01.702519 (+ 486us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:01.702546 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:02:01.703017 (+ 471us) log.cc:842] Serialized 52 byte log entry
0807 08:02:01.914451 (+211434us) raft_consensus.cc:1631] Marking committed up to 8
0807 08:02:02.152161 (+237710us) log.cc:842] Serialized 14 byte log entry
0807 08:02:02.156461 (+ 4300us) log.cc:842] Serialized 14 byte log entry
0807 08:02:02.657892 (+501431us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b1800149140. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:02:03.295010 (+637118us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b1800149140. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b19fba4ca 00007f6b2369bf99 00007f6b19e7e990 00007f6b1a040864 00007f6b1a040155 00007f6b1cf633d0 00007f6b1cf75576 00007f6b1cf860d7 00007f6b1cf86058
0807 08:02:03.295206 (+ 196us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:03.295277 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:03.295324 (+ 47us) raft_consensus.cc:1671] finished
0807 08:02:03.295336 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:03.295470 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":679331,"prepare.queue_time_us":69659,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":82,"raft.queue_time_us":1145538,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":274,"spinlock_wait_cycles":168534784,"thread_start_us":390041,"threads_started":2,"wal-append.queue_time_us":326766,"wal-append.run_cpu_time_us":3483,"wal-append.run_wall_time_us":1270614}
I20260807 08:02:03.346266 13808 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:03.336014 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:03.364665 13817 raft_consensus.cc:3037] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 3 FOLLOWER]: Advancing to term 5
W20260807 08:02:03.348611 13508 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1588644cd4f943a7b5d776326af48218, TS 9210dbc4cc9c45e5b26a463b705d362c -> TS 22ff0a1f4a4241d794ab182432c85893 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:03.365365 13508 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1588644cd4f943a7b5d776326af48218, TS 9210dbc4cc9c45e5b26a463b705d362c -> TS 22ff0a1f4a4241d794ab182432c85893 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.349359 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.363924 14627 raft_consensus.cc:3037] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:03.367586 14626 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:02:03.359102 14583 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:03.369163 13537 leader_election.cc:302] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:03.373950 13809 raft_consensus.cc:2396] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260807 08:02:03.374027 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.374421 14623 raft_consensus.cc:2454] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:02:03.375617 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.379436 13590 raft_consensus.cc:1226] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:03.385779 14623 raft_consensus.cc:2354] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9.
I20260807 08:02:03.393424 14627 raft_consensus.cc:2454] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:02:03.394901 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.395337 13812 raft_consensus.cc:1261] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260807 08:02:03.395881 13812 pending_rounds.cc:76] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 2
I20260807 08:02:03.396144 13812 pending_rounds.cc:97] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260807 08:02:03.396445 13812 raft_consensus.cc:2866] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:02:03.393206 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.391976 13537 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:03.399231 14629 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
mode: GRACEFUL
new_leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
from {username='slave'} at 127.0.0.1:55594
I20260807 08:02:03.399525 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:03.399717 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:03.399869 14629 raft_consensus.cc:595] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Received request to transfer leadership to TS 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:03.305938 18423 raft_consensus.cc:686] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 9 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:03.400323 14629 raft_consensus.cc:603] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:02:03.401499 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.402398 13817 raft_consensus.cc:2454] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:02:03.357157 18482 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:03.412966 13590 raft_consensus.cc:1226] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 3. Current term is 4. Ops: []
I20260807 08:02:03.415187 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.376904 18695 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:02:03.407935 13584 raft_consensus.cc:1261] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260807 08:02:03.410197 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.411046 13540 leader_election.cc:302] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:03.427453 13537 leader_election.cc:302] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:03.416747 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.427474 13584 pending_rounds.cc:76] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: Aborting all ops after (but not including) 7
I20260807 08:02:03.405612 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.428407 13584 pending_rounds.cc:97] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260807 08:02:03.428681 13584 raft_consensus.cc:2866] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:02:03.371426 17943 leader_election.cc:288] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:03.417925 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:03.400712 18432 raft_consensus.cc:2781] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:02:03.420881 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.421213 18482 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.377143 18308 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260807 08:02:03.429311 13508 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6f5ccbaa50ae4451a35038284693254c, TS 9210dbc4cc9c45e5b26a463b705d362c -> TS 22ff0a1f4a4241d794ab182432c85893 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.429505 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
W20260807 08:02:03.478370 13508 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6f5ccbaa50ae4451a35038284693254c, TS 9210dbc4cc9c45e5b26a463b705d362c -> TS 22ff0a1f4a4241d794ab182432c85893 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.414532 18634 raft_consensus.cc:2781] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:02:03.433924 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.386986 18646 raft_consensus.cc:2781] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:02:03.484472 14619 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
W20260807 08:02:03.478552 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 48592.6us
I20260807 08:02:03.489698 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:03.498338 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.491753 13584 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:02:03.500615 14583 leader_election.cc:302] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:03.502240 14583 leader_election.cc:302] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:03.503487 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.506069 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.506819 14629 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:03.505259 14627 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 10 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260807 08:02:03.431336 18695 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.433853 18521 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.458596 18681 raft_consensus.cc:482] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.484560 18308 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:03.482511 18376 raft_consensus.cc:2726] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:03.484622 18425 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:03.491973 18634 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.477758 18585 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:03.397185 18592 consensus_queue.cc:260] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.549853 18590 raft_consensus.cc:2781] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:02:03.549649 18646 raft_consensus.cc:686] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [term 8 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:03.590117 18308 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Advancing to term 11
I20260807 08:02:03.548997 18584 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.076s
I20260807 08:02:03.578174 18695 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:03.641739 13706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1648) took 1105 ms. Trace:
I20260807 08:02:03.704118 13706 rpcz_store.cc:276] 0807 08:02:02.536468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:02.536610 (+ 142us) service_pool.cc:227] Handling call
0807 08:02:03.641526 (+1104916us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b54002b9654. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:03.641680 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86453376}
I20260807 08:02:03.705914 13589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:56576 (request call id 1030) took 1531 ms. Trace:
I20260807 08:02:03.706372 13589 rpcz_store.cc:276] 0807 08:02:02.174842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:02.175031 (+ 189us) service_pool.cc:227] Handling call
0807 08:02:03.694008 (+1518977us) spinlock_profiling.cc:243] Waited 48.9 ms on lock 0x7b54003d0ed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf717c5 00007f6b1cf69c1a 00007f6b21ead6d9 00007f6b1b597184 00007f6b1b597112 00007f6b1b597041 00007f6b1b596fbc 00007f6b1b596272 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:03.705810 (+ 11802us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112545152}
I20260807 08:02:03.710086 13808 raft_consensus.cc:1261] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260807 08:02:03.577741 18432 raft_consensus.cc:686] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
W20260807 08:02:03.711359 18713 log.cc:924] Time spent T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260807 08:02:03.711447 18432 consensus_queue.cc:237] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.714162 18482 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.714233 18592 raft_consensus.cc:3037] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:03.687816 14629 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:02:03.588438 18445 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:03.612315 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.614866 18634 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:03.700765 14618 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 4.
I20260807 08:02:03.590880 18521 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:03.710244 18710 log.cc:924] Time spent T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.225s user 0.000s sys 0.000s
I20260807 08:02:03.608258 18681 raft_consensus.cc:504] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.717036 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.613494 18423 consensus_queue.cc:237] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.730262 18521 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.728776 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.732551 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dfc89fb081405f96a0b53d76aeea38" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:03.733757 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.733968 18681 leader_election.cc:288] T 38e7d8fef17f4bdb92d0c21739339421 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.734411 14618 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 9.
I20260807 08:02:03.735381 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.736004 14618 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:02:03.615008 18425 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.739812 18425 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:03.745638 18445 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.747141 13585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33050 (request call id 690) took 1649 ms. Trace:
I20260807 08:02:03.747638 13585 rpcz_store.cc:276] 0807 08:02:02.098068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:02.098241 (+ 173us) service_pool.cc:227] Handling call
0807 08:02:03.746901 (+1648660us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54002053d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:03.747071 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":393632512}
I20260807 08:02:03.748131 18445 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:03.746194 13587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 1989) took 1918 ms (client timeout 1627 ms). Trace:
W20260807 08:02:03.749569 18689 log.cc:924] Time spent T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.628s user 0.006s sys 0.030s
I20260807 08:02:03.749166 18646 consensus_queue.cc:237] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:03.749730 13587 rpcz_store.cc:269] 0807 08:02:01.827913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:01.832466 (+ 4553us) service_pool.cc:227] Handling call
0807 08:02:03.745954 (+1913488us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54001356d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:03.746100 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":254580096}
I20260807 08:02:03.689414 18590 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.736717 18345 raft_consensus.cc:504] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.718219 18344 raft_consensus.cc:2741] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260807 08:02:03.773268 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:02:03.793839 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 289277us
I20260807 08:02:03.667215 18585 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.765802 18590 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:02:03.797101 14629 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434"
dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
mode: GRACEFUL
new_leader_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
from {username='slave'} at 127.0.0.1:55594
I20260807 08:02:03.812417 14629 raft_consensus.cc:595] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Received request to transfer leadership to TS 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:03.812875 14629 raft_consensus.cc:603] T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260807 08:02:03.803426 14627 raft_consensus.cc:2354] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 5. Current term is 6.
I20260807 08:02:03.804119 18113 consensus_queue.cc:1035] T a01b1964575c4708bf9760dff95db7f3 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:02:03.804548 18345 leader_election.cc:288] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:03.823944 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 406217us
W20260807 08:02:03.829378 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 418423us
I20260807 08:02:03.830406 13540 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:03.841318 11825 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260807 08:02:03.841755 13540 leader_election.cc:302] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:03.849079 13540 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:03.799196 18425 consensus_queue.cc:1046] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260807 08:02:03.852020 13540 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:03.847604 18482 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:02:03.881541 13817 raft_consensus.cc:1261] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 08:02:03.870517 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
W20260807 08:02:03.854300 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 362723us
I20260807 08:02:03.885715 13816 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:03.851421 18398 raft_consensus.cc:2726] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 9b4b1bd6a74d452e860b1420b0ef1434 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 7. Current term is 8.
I20260807 08:02:03.850821 18726 raft_consensus.cc:2781] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election won for term 8
I20260807 08:02:03.853278 18436 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:03.895439 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:03.857987 18731 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:02:03.878458 18482 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:03.832293 18695 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.914505 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:03.919250 13804 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:03.878587 18729 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 08:02:03.891788 18732 raft_consensus.cc:3032] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:02:03.921108 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.931849 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:03.882170 18577 consensus_queue.cc:1035] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260807 08:02:03.930742 13811 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:02:03.934194 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:03.936380 13585 raft_consensus.cc:1261] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260807 08:02:03.941289 13579 raft_consensus.cc:1261] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260807 08:02:03.847854 18539 consensus_queue.cc:260] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.939471 14629 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260807 08:02:03.913102 18726 raft_consensus.cc:686] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [term 8 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:03.919183 18482 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:03.890852 18303 consensus_queue.cc:1046] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260807 08:02:03.957288 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:03.958760 13587 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:03.924081 18732 raft_consensus.cc:729] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:03.900975 18436 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.935917 18746 consensus_queue.cc:1035] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260807 08:02:03.939857 18607 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:03.954118 18729 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:03.973901 18695 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:03.977622 18647 raft_consensus.cc:3032] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:02:03.979889 13700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1649) took 1433 ms. Trace:
I20260807 08:02:03.980342 13700 rpcz_store.cc:276] 0807 08:02:02.546495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:02.546640 (+ 145us) service_pool.cc:227] Handling call
0807 08:02:03.979692 (+1433052us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b54001dc4d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:03.979822 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68350848}
W20260807 08:02:04.010216 18739 log.cc:924] Time spent T e5ef275b7c0a45619cbc05266aea24dc P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260807 08:02:04.010142 18732 consensus_queue.cc:260] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:03.957834 18731 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.012634 18731 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:03.962036 18489 raft_consensus.cc:2726] T b09437796639470caac9548c71d72fb2 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:04.013005 18726 consensus_queue.cc:237] T 0e72e5415e504959b4763e580400f8ee P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.014741 18731 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:04.015357 18729 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.009433 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54072 (request call id 1712) took 1294 ms. Trace:
I20260807 08:02:03.978055 18647 raft_consensus.cc:729] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:04.020123 13805 rpcz_store.cc:276] 0807 08:02:02.685300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:02.685464 (+ 164us) service_pool.cc:227] Handling call
0807 08:02:03.979266 (+1293802us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b540027bad4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:03.979419 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57018112}
I20260807 08:02:03.972342 18607 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:04.015527 18308 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:04.021009 14625 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260807 08:02:03.965245 18436 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:04.010206 18743 log.cc:924] Time spent T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20260807 08:02:04.036195 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 148385us
I20260807 08:02:04.035059 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:04.039769 14583 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:04.045367 14583 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893; no voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:04.043255 14629 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:02:04.048868 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:04.049523 13579 raft_consensus.cc:2454] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:04.015997 18634 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.051467 18584 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:04.051931 18584 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:04.052296 18584 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:04.052896 18634 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.053622 18303 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260807 08:02:04.055145 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:03.978879 18585 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:04.057297 13537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 184538us
I20260807 08:02:04.058741 13804 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:02:04.047204 13769 leader_election.cc:302] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:04.060968 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:04.038064 18539 raft_consensus.cc:3037] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:04.065763 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:04.017577 18729 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.067303 13584 raft_consensus.cc:1226] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 9. Current term is 10. Ops: [7.8-9.9]
I20260807 08:02:04.043643 18732 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:04.069171 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:04.070981 13589 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:04.071260 13540 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:02:04.060160 13418 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 0248ade936924e628c46b8de6252af1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 77da774f2bb54cf28302d53d0db224ad is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } } }
W20260807 08:02:04.073141 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 110588us
I20260807 08:02:04.074301 14580 leader_election.cc:302] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:04.068579 13769 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:04.062301 18514 raft_consensus.cc:2726] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:04.078845 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:04.063627 18410 raft_consensus.cc:2726] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 08:02:04.065274 18592 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:04.064630 18593 raft_consensus.cc:2726] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:02:04.104789 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:04.104549 18592 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.108654 18592 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.032580 18647 consensus_queue.cc:260] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.109869 11825 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260807 08:02:04.091472 18729 raft_consensus.cc:2781] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:02:04.110522 18729 raft_consensus.cc:482] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:04.110822 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:02:04.118008 13540 leader_election.cc:302] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:04.094645 13584 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 9. Current term is 10. Ops: []
I20260807 08:02:04.120525 18762 consensus_queue.cc:1035] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260807 08:02:04.121950 18581 leader_election.cc:288] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:04.064725 18745 consensus_queue.cc:1035] T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260807 08:02:04.124037 18308 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 11 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:04.124682 18584 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.127929 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:04.128425 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:04.129190 13817 raft_consensus.cc:3037] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Advancing to term 11
I20260807 08:02:04.103353 13589 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 9.
I20260807 08:02:04.142361 13587 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 8. Current term is 9. Ops: []
I20260807 08:02:04.077684 18575 consensus_queue.cc:1046] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260807 08:02:04.117755 18681 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:04.129144 18760 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:04.110869 18729 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:04.161919 13769 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:04.162945 13769 leader_election.cc:302] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:04.134593 18303 raft_consensus.cc:2726] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260807 08:02:04.093582 18583 raft_consensus.cc:2781] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader election won for term 6
I20260807 08:02:04.168614 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1643) took 2109 ms. Trace:
I20260807 08:02:04.115037 18577 consensus_queue.cc:1046] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260807 08:02:04.169720 13590 raft_consensus.cc:1226] T 06a735af95cd49e1942d669d2e5d91c6 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 5. Current term is 6. Ops: []
W20260807 08:02:04.159744 18759 log.cc:924] Time spent T 74ccb94230614cc48e846580ad82e994 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.068s user 0.001s sys 0.001s
I20260807 08:02:04.152371 13817 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 11.
I20260807 08:02:04.169011 13702 rpcz_store.cc:276] 0807 08:02:02.059200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:02.224169 (+164969us) service_pool.cc:227] Handling call
0807 08:02:02.242133 (+ 17964us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:02.242481 (+ 348us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:02.242513 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:02.265448 (+ 22935us) log.cc:842] Serialized 26 byte log entry
0807 08:02:02.408488 (+143040us) raft_consensus.cc:1631] Marking committed up to 5
0807 08:02:02.434339 (+ 25851us) log.cc:842] Serialized 14 byte log entry
0807 08:02:02.435367 (+ 1028us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:02.435448 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:02.574046 (+138598us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x7b1800403c00. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b22ee28b1 00007f6b22eef661 00007f6b1a043638 00007f6b1a046b11 00007f6b1a046ac9 00007f6b1a046a59 00007f6b1a046a21 00007f6b1a045d1d 00007f6b232a7204 00007f6b232a7039 00007f6b1a025dc6
0807 08:02:04.134094 (+1560048us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b2400418268. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19e9bcf4 00007f6b19e99dfb 00007f6b19f28559 00007f6b19f281e5 00007f6b19f27740 00007f6b1cc1a578 00007f6b1cc1acd8 00007f6b1cbf26e8 00007f6b1cbedaea 00007f6b1cbecf36
0807 08:02:04.168412 (+ 34318us) raft_consensus.cc:1671] finished
0807 08:02:04.168434 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:04.168554 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":798,"prepare.queue_time_us":27024,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":56431,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":5715,"spinlock_wait_cycles":115561728,"thread_start_us":88578,"threads_started":1,"wal-append.queue_time_us":137412}
I20260807 08:02:04.171392 13769 leader_election.cc:302] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:04.177065 13537 leader_election.cc:302] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:04.166349 18380 raft_consensus.cc:2781] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:04.158284 18681 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.182893 14623 raft_consensus.cc:1226] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [term 9 LEADER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260807 08:02:04.183210 13589 raft_consensus.cc:1261] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260807 08:02:04.183279 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 109031us
I20260807 08:02:04.164150 18748 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:04.166518 18592 raft_consensus.cc:2781] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 08:02:04.188558 14619 raft_consensus.cc:3032] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 08:02:04.188716 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:04.189936 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:04.192726 14619 raft_consensus.cc:729] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:04.166262 18767 consensus_queue.cc:1035] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
I20260807 08:02:04.197978 14619 consensus_queue.cc:260] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.163899 18542 consensus_queue.cc:1035] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260807 08:02:04.168543 18585 consensus_queue.cc:1046] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.028s
I20260807 08:02:04.167410 18760 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:04.202472 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:04.184279 18577 consensus_queue.cc:1225] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer 7a889001d6494fe5b7cbbe7da1dae302 log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260807 08:02:04.639145 18760 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:04.639959 18760 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.593629 18268 raft_consensus.cc:3032] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:02:04.192281 18731 raft_consensus.cc:2781] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 11 FOLLOWER]: Leader election won for term 11
I20260807 08:02:04.196591 18592 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:04.637650 18745 raft_consensus.cc:482] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:02:04.640182 18702 log.cc:924] Time spent T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 1.449s user 0.000s sys 0.015s
W20260807 08:02:04.309616 18641 log.cc:924] Time spent T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260807 08:02:04.243136 18581 raft_consensus.cc:482] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
W20260807 08:02:04.241528 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 51771.5us
I20260807 08:02:04.187402 18695 raft_consensus.cc:2781] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 08:02:04.244737 18514 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.398s
I20260807 08:02:04.644398 14629 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260807 08:02:04.637112 14619 raft_consensus.cc:3037] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:04.191373 18681 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:04.183126 18380 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
W20260807 08:02:04.598100 18590 consensus_meta.cc:287] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.756s user 0.011s sys 0.001s
W20260807 08:02:04.514757 18689 log.cc:924] Time spent T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.209s user 0.000s sys 0.000s
I20260807 08:02:04.192078 18583 raft_consensus.cc:686] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:04.537827 18762 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:04.659217 18592 raft_consensus.cc:3037] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:04.289415 18376 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.664423 18585 raft_consensus.cc:3032] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 08:02:04.186605 18741 consensus_queue.cc:1046] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.486s
I20260807 08:02:04.242862 18308 raft_consensus.cc:482] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.385708 18646 raft_consensus.cc:482] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.640655 18760 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:04.631093 18774 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.646812 18268 raft_consensus.cc:729] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
W20260807 08:02:04.727861 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 48986.2us
I20260807 08:02:04.674307 18731 raft_consensus.cc:686] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [term 11 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:04.521778 18584 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:04.725025 18772 log.cc:924] Time spent T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.228s user 0.000s sys 0.001s
I20260807 08:02:04.721117 18380 raft_consensus.cc:3037] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:04.725222 18782 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:04.663468 18745 raft_consensus.cc:504] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:04.733891 18585 raft_consensus.cc:729] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:04.203635 18748 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:04.793542 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 550306us
I20260807 08:02:04.795317 14629 raft_consensus.cc:1226] T 4614fbd9fc524f61bb9d11ea324e783e P 22ff0a1f4a4241d794ab182432c85893 [term 10 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260807 08:02:04.797642 18778 log.cc:924] Time spent T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20260807 08:02:04.696834 18581 raft_consensus.cc:504] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.727872 18590 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.749960 18376 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:04.774384 18695 raft_consensus.cc:686] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:04.770844 18785 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.790246 18542 raft_consensus.cc:482] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.754596 18634 consensus_queue.cc:1035] T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.046s
I20260807 08:02:04.754572 18308 raft_consensus.cc:504] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.762961 18646 raft_consensus.cc:504] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.733487 18762 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:04.854449 18776 log.cc:924] Time spent T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260807 08:02:04.779398 18774 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.789402 18793 consensus_queue.cc:1035] T b17c7c715cf146728d57366f8e070d31 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
W20260807 08:02:04.866798 18482 consensus_meta.cc:287] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.900s user 0.008s sys 0.004s
I20260807 08:02:04.867342 18482 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.869081 18731 consensus_queue.cc:237] T d6afc407b1b94edb94f8e7e8624c9221 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:04.871176 18376 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:04.827962 18785 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:04.874955 18785 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:04.877424 14626 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 663) took 1477 ms (client timeout 1693 ms). Trace:
I20260807 08:02:04.877952 18774 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:04.877838 14626 rpcz_store.cc:269] 0807 08:02:03.399845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:03.400793 (+ 948us) service_pool.cc:227] Handling call
0807 08:02:04.877245 (+1476452us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b5400073354. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:04.877364 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1104747392}
W20260807 08:02:04.883996 18729 consensus_meta.cc:287] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.714s user 0.010s sys 0.000s
I20260807 08:02:04.865300 18803 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:04.863168 18784 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.830813 18542 raft_consensus.cc:504] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:04.798694 18782 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.903483 18745 leader_election.cc:288] T 88e3dc2093eb4153a975824017d2aafb P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:04.905720 18755 log.cc:924] Time spent T 04e678279be64e6ca8a141d6b761e522 P 77da774f2bb54cf28302d53d0db224ad: Append to log took a long time: real 0.906s user 0.002s sys 0.001s
I20260807 08:02:04.810748 18583 consensus_queue.cc:237] T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.883987 18774 raft_consensus.cc:482] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:04.899201 18781 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:04.830041 18268 consensus_queue.cc:260] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.916916 18748 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:04.900045 18729 raft_consensus.cc:504] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.897917 18695 consensus_queue.cc:237] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:04.912262 18607 consensus_meta.cc:287] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.880s user 0.012s sys 0.000s
I20260807 08:02:04.918159 18803 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:04.922430 18581 leader_election.cc:288] T b74446812e814c7e9b4a1bd80bfb485d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.924667 18646 leader_election.cc:288] T 74ccb94230614cc48e846580ad82e994 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.860088 18585 consensus_queue.cc:260] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.926460 18762 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.968489 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:04.969327 14623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 661) took 1595 ms. Trace:
I20260807 08:02:04.969434 13585 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:04.969784 14623 rpcz_store.cc:276] 0807 08:02:03.373814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:03.548560 (+174746us) service_pool.cc:227] Handling call
0807 08:02:03.610035 (+ 61475us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:04.190466 (+580431us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:04.969064 (+778598us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b54004b32d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:04.969258 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1316802304}
I20260807 08:02:04.970484 18585 raft_consensus.cc:3037] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:04.971866 13811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59332 (request call id 758) took 1616 ms. Trace:
I20260807 08:02:04.972223 13811 rpcz_store.cc:276] 0807 08:02:03.355731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:03.360982 (+ 5251us) service_pool.cc:227] Handling call
0807 08:02:04.971681 (+1610699us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54001fc2d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:04.971787 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1142230912}
I20260807 08:02:04.974908 18803 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:04.975618 18308 leader_election.cc:288] T e110e8648fd04684ae2bf4d0c71752a3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.980429 18681 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.985668 13585 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:02:04.985976 18590 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.987203 18482 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.987403 18681 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:04.991695 13585 raft_consensus.cc:1226] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:04.991815 13430 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57868 (request call id 216) took 10230 ms. Trace:
I20260807 08:02:04.966320 18607 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:04.965713 18760 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.007580 18807 raft_consensus.cc:482] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:05.008234 18807 raft_consensus.cc:504] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.010548 18807 leader_election.cc:288] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:04.990150 18268 raft_consensus.cc:3037] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:04.990243 18482 raft_consensus.cc:482] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:04.992090 18590 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:04.926517 18774 raft_consensus.cc:504] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.019809 13580 raft_consensus.cc:1226] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 8. Current term is 9. Ops: []
I20260807 08:02:04.987987 18781 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.022132 18592 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.024111 18592 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:04.992365 13430 rpcz_store.cc:276] 0807 08:01:54.761695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:01:54.761933 (+ 238us) service_pool.cc:227] Handling call
0807 08:02:04.991722 (+10229789us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 08:02:01.096285 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0807 08:02:01.263188 (+166903us) write_op.cc:432] Acquiring schema lock in shared mode
0807 08:02:01.337177 (+ 73989us) write_op.cc:435] Acquired schema lock
0807 08:02:01.364710 (+ 27533us) tablet.cc:660] Decoding operations
0807 08:02:01.537676 (+172966us) write_op.cc:620] Acquiring the partition lock for write op
0807 08:02:01.567847 (+ 30171us) write_op.cc:641] Partition lock acquired for write op
0807 08:02:01.567893 (+ 46us) tablet.cc:683] Acquiring locks for 5 operations
0807 08:02:01.690241 (+122348us) tablet.cc:699] Row locks acquired
0807 08:02:01.690277 (+ 36us) write_op.cc:260] PREPARE: finished
0807 08:02:01.690436 (+ 159us) write_op.cc:270] Start()
0807 08:02:01.690576 (+ 140us) write_op.cc:276] Timestamp: P: 1786089721690379 usec, L: 0
0807 08:02:01.690607 (+ 31us) op_driver.cc:331] REPLICATION: starting
0807 08:02:01.694119 (+ 3512us) log.cc:842] Serialized 2069 byte log entry
0807 08:02:02.794332 (+1100213us) op_driver.cc:444] REPLICATION: finished
0807 08:02:02.990228 (+195896us) write_op.cc:301] APPLY: starting
0807 08:02:02.990306 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0807 08:02:02.990449 (+ 143us) tablet.cc:1368] finished BulkCheckPresence
0807 08:02:02.990533 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 08:02:02.991102 (+ 569us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 08:02:02.991352 (+ 250us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0807 08:02:02.991378 (+ 26us) write_op.cc:312] APPLY: finished
0807 08:02:02.991601 (+ 223us) log.cc:842] Serialized 47 byte log entry
0807 08:02:02.991790 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:02:02.991893 (+ 103us) write_op.cc:454] Released schema lock
0807 08:02:02.991946 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0807 08:02:02.992283 (+ 337us) write_op.cc:489] Releasing partition, row and schema locks
0807 08:02:02.992318 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":195620,"apply.run_cpu_time_us":2371,"apply.run_wall_time_us":2370,"num_ops":5,"prepare.queue_time_us":1380191,"prepare.run_cpu_time_us":32782,"prepare.run_wall_time_us":905762,"raft.queue_time_us":155709,"raft.run_cpu_time_us":73744,"raft.run_wall_time_us":127249,"replication_time_us":1103625,"thread_start_us":1503577,"threads_started":4,"wal-append.queue_time_us":620169,"wal-append.run_cpu_time_us":45690,"wal-append.run_wall_time_us":1679969}]]}
I20260807 08:02:05.028522 18542 leader_election.cc:288] T 6f5ccbaa50ae4451a35038284693254c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:05.030357 18378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260807 08:02:04.915095 18784 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.075821 18774 leader_election.cc:288] T cc32413495ee41909c7c5494c8095084 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:05.078331 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 1.87094e+06us
I20260807 08:02:05.081262 18709 raft_consensus.cc:2781] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:02:05.081976 18709 raft_consensus.cc:686] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 LEADER]: Becoming Leader. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
W20260807 08:02:05.081377 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 1.82721e+06us
I20260807 08:02:05.018896 18482 raft_consensus.cc:504] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.050860 18544 raft_consensus.cc:3032] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 9 LEADER]: Stepping down as leader of term 9
I20260807 08:02:05.089964 18544 raft_consensus.cc:729] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:05.090791 18544 consensus_queue.cc:260] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:05.056994 18756 log.cc:924] Time spent T 232127da9f574f41a5a8087ae775ef6b P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 1.074s user 0.003s sys 0.000s
I20260807 08:02:05.085743 18782 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:05.093969 18230 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:05.094391 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39150 (request call id 1887) took 1957 ms. Trace:
I20260807 08:02:05.087447 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:05.094944 13711 rpcz_store.cc:276] 0807 08:02:03.137680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:03.137812 (+ 132us) service_pool.cc:227] Handling call
0807 08:02:03.138283 (+ 471us) raft_consensus.cc:1457] Updating replica for 3 ops
0807 08:02:03.160765 (+ 22482us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:03.160797 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0807 08:02:03.164575 (+ 3778us) log.cc:842] Serialized 78 byte log entry
0807 08:02:03.283671 (+119096us) raft_consensus.cc:1631] Marking committed up to 3
0807 08:02:03.426571 (+142900us) log.cc:842] Serialized 14 byte log entry
0807 08:02:05.094072 (+1667501us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:05.094148 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:05.094191 (+ 43us) raft_consensus.cc:1671] finished
0807 08:02:05.094204 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:05.094335 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1679971,"prepare.queue_time_us":314759,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":23495,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":7778,"spinlock_wait_cycles":157568,"thread_start_us":213670,"threads_started":2,"wal-append.queue_time_us":226346,"wal-append.run_cpu_time_us":9649,"wal-append.run_wall_time_us":1348367}
I20260807 08:02:05.082826 18709 consensus_queue.cc:237] T e67bd63471c94c19a092c8f1aef64408 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.088822 18496 raft_consensus.cc:482] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.089268 18419 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.081262 18478 raft_consensus.cc:3032] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:02:05.148134 18478 raft_consensus.cc:729] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:05.149061 18478 consensus_queue.cc:260] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.101244 18647 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:05.026811 18590 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.083492 18729 leader_election.cc:288] T 966e99d2078c4b1c9fc921c1666b45a2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:02:05.107668 18780 log.cc:924] Time spent T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260807 08:02:05.109814 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:05.111737 18781 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.094455 18230 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.154745 13706 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:05.091809 18544 raft_consensus.cc:3037] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:05.166417 13708 raft_consensus.cc:1261] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260807 08:02:05.105702 18539 consensus_meta.cc:287] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 1.036s user 0.006s sys 0.006s
I20260807 08:02:05.200923 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67bd63471c94c19a092c8f1aef64408" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:02:05.204243 13592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56592 (request call id 2016) took 2346 ms (client timeout 2230 ms). Trace:
W20260807 08:02:05.204777 13592 rpcz_store.cc:269] 0807 08:02:02.858175 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:02.910436 (+ 52261us) service_pool.cc:227] Handling call
0807 08:02:05.204025 (+2293589us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b5400205dd4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:05.204166 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":870317952}
I20260807 08:02:05.208376 18404 raft_consensus.cc:482] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.218827 13702 raft_consensus.cc:1261] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260807 08:02:05.224952 13700 raft_consensus.cc:1226] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20260807 08:02:05.200639 18732 consensus_meta.cc:287] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 1.111s user 0.013s sys 0.000s
I20260807 08:02:05.231765 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d5ddf3f7d84e0ebda02d99abff4f0e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.178550 17943 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.178231 18370 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:02:05.235886 13657 connection.cc:232] RPC call timeout handler was delayed by -1.0149096499662846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
I20260807 08:02:05.237004 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44a80bdfaf145389eacef429f26eedc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:05.245913 13701 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:05.246491 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:02:05.256315 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 305180us
I20260807 08:02:05.258402 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:05.231442 18404 raft_consensus.cc:504] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.259898 18766 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.261427 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.262773 13708 raft_consensus.cc:2396] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260807 08:02:05.259192 13592 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:02:05.217227 18645 raft_consensus.cc:482] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.232374 18402 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.165385 18496 raft_consensus.cc:504] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.232097 17585 raft_consensus.cc:482] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.148180 18419 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.238046 18658 raft_consensus.cc:482] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:05.238046 18654 raft_consensus.cc:482] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:05.375375 13701 raft_consensus.cc:2454] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 10.
I20260807 08:02:05.420961 13710 raft_consensus.cc:1261] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260807 08:02:05.328199 18664 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.747s user 0.017s sys 0.039s
I20260807 08:02:05.260435 18766 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.261822 18370 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.379694 18774 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.365772 18732 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:05.261325 17943 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.317077 18402 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.305642 18645 raft_consensus.cc:504] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
W20260807 08:02:05.550683 18723 log.cc:924] Time spent T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 1.488s user 0.000s sys 0.004s
I20260807 08:02:05.555634 18766 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:05.526925 18585 consensus_meta.cc:287] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.536s user 0.002s sys 0.008s
W20260807 08:02:05.498992 18664 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.921s user 0.018s sys 0.039s
I20260807 08:02:05.568178 18634 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:05.333534 17585 raft_consensus.cc:504] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.364113 18655 consensus_queue.cc:1046] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.181s
I20260807 08:02:05.364974 18658 raft_consensus.cc:504] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.583019 13814 raft_consensus.cc:1226] T 13ee9b6f7fa547a3badcb0eba0210119 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 LEADER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 9. Current term is 10. Ops: []
I20260807 08:02:05.578770 13657 leader_election.cc:302] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:02:05.579317 18583 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.009s sys 0.006s
I20260807 08:02:05.580211 18607 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.558374 18781 raft_consensus.cc:482] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:05.584234 18539 consensus_queue.cc:1046] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260807 08:02:05.591614 18370 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.549090 18774 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.592747 18496 leader_election.cc:288] T c8727dbda122418487cb692370c2399e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.593909 14620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 631) took 2271 ms. Trace:
I20260807 08:02:05.594516 18774 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:05.581698 18380 consensus_meta.cc:287] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302: Time spent flushing consensus metadata: real 0.768s user 0.005s sys 0.005s
I20260807 08:02:05.558722 18732 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.595445 18380 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.581418 18585 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:05.597803 18732 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.584861 18634 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.598248 18585 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.591260 18781 raft_consensus.cc:504] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.587086 18268 raft_consensus.cc:482] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.380550 18654 raft_consensus.cc:504] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.594280 14620 rpcz_store.cc:276] 0807 08:02:03.322887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:03.334003 (+ 11116us) service_pool.cc:227] Handling call
0807 08:02:03.346836 (+ 12833us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:03.347270 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:03.347300 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:03.378109 (+ 30809us) log.cc:842] Serialized 26 byte log entry
0807 08:02:03.828175 (+450066us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:02:03.828230 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:03.828318 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:05.217295 (+1388977us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b44000232cc. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b19f55a95 00007f6b19f564b2 00007f6b1cbfe511 00007f6b1cbf9e1a 00007f6b1cbedbee 00007f6b1cbecf36 00007f6b1cbecb58 00007f6b1cbf6551 00007f6b1cbf6509 00007f6b1cbf6499
0807 08:02:05.593590 (+376295us) raft_consensus.cc:1671] finished
0807 08:02:05.593611 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:05.593826 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15961,"raft.queue_time_us":9451,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":15304,"spinlock_wait_cycles":9276672,"thread_start_us":362572,"threads_started":1,"wal-append.queue_time_us":485356}
I20260807 08:02:05.601058 18634 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.593272 18496 raft_consensus.cc:2781] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 08:02:05.610644 13709 raft_consensus.cc:1226] T d44a80bdfaf145389eacef429f26eedc P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 9. Current term is 10. Ops: [6.3-9.5]
I20260807 08:02:05.610025 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.619534 13700 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:02:05.618438 18496 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:05.624042 18419 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.624486 18496 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:05.626760 18402 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.628319 17943 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.629755 18781 leader_election.cc:288] T a01b1964575c4708bf9760dff95db7f3 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.588045 18376 raft_consensus.cc:482] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.631088 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.632611 13709 raft_consensus.cc:2379] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:05.633505 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:02:05.589936 18583 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s user 0.010s sys 0.006s
I20260807 08:02:05.615084 18585 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.630052 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:02:05.644990 14627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42252 (request call id 669) took 2030 ms (client timeout 1583 ms). Trace:
W20260807 08:02:05.645613 14627 rpcz_store.cc:269] 0807 08:02:03.614554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:03.716725 (+102171us) service_pool.cc:227] Handling call
0807 08:02:05.644711 (+1927986us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b54004deed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:05.644896 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":172902784}
I20260807 08:02:05.621640 18658 leader_election.cc:288] T 8a8da02e8f51481cbcca504f8cff6f80 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.637961 13709 raft_consensus.cc:2379] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 11: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:05.660354 13709 raft_consensus.cc:1261] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260807 08:02:05.617295 18268 raft_consensus.cc:504] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.663586 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.665071 13709 raft_consensus.cc:1261] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260807 08:02:05.701272 18816 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.701965 18816 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.705382 18816 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.634536 18376 raft_consensus.cc:504] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.651311 18590 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.672412 18380 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:05.740703 18496 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.733047 18489 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.746820 18376 leader_election.cc:288] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.726605 18658 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:05.749953 18489 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.751747 18658 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.758949 18489 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
W20260807 08:02:05.763836 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 181899us
W20260807 08:02:05.771975 13711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.003s sys 0.009s
W20260807 08:02:05.772320 13711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s user 0.003s sys 0.009s
W20260807 08:02:05.770967 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 192287us
W20260807 08:02:05.775020 13661 connection.cc:232] RPC call timeout handler was delayed by -1.0027817219961435s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1693s for a response
I20260807 08:02:05.768183 18591 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:02:05.773584 18695 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.043s sys 0.069s
W20260807 08:02:05.775271 18647 consensus_meta.cc:287] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.625s user 0.008s sys 0.005s
I20260807 08:02:05.791813 18376 raft_consensus.cc:482] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.771618 18748 raft_consensus.cc:482] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.813709 18591 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.816000 18591 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
W20260807 08:02:05.799435 18544 consensus_meta.cc:287] T 4614fbd9fc524f61bb9d11ea324e783e P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.585s user 0.010s sys 0.000s
I20260807 08:02:05.818835 13580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56594 (request call id 1901) took 1885 ms. Trace:
I20260807 08:02:05.819243 13580 rpcz_store.cc:276] 0807 08:02:03.933244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:03.933430 (+ 186us) service_pool.cc:227] Handling call
0807 08:02:03.957396 (+ 23966us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:05.021201 (+1063805us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:05.818556 (+797355us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54003d0ed4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:05.818759 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2442911616}
I20260807 08:02:05.821749 13705 raft_consensus.cc:1261] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260807 08:02:05.786957 18658 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.810134 18461 raft_consensus.cc:482] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.773412 18539 consensus_queue.cc:1225] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer 7a889001d6494fe5b7cbbe7da1dae302 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260807 08:02:05.792459 18496 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 election: Requested vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.793591 18268 leader_election.cc:288] T 13ee9b6f7fa547a3badcb0eba0210119 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.778319 18645 leader_election.cc:288] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.793035 18827 raft_consensus.cc:482] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.796296 17943 raft_consensus.cc:482] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:02:05.822043 18822 log.cc:924] Time spent T d37920c01cb94122bd1c5a0bb331be31 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
W20260807 08:02:05.809305 18695 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s user 0.043s sys 0.069s
I20260807 08:02:05.795356 18760 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.810556 18654 leader_election.cc:288] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.811175 18655 raft_consensus.cc:482] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:05.850889 14583 leader_election.cc:302] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:05.852962 13817 raft_consensus.cc:1226] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 LEADER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 8. Current term is 9. Ops: []
I20260807 08:02:05.842276 18801 raft_consensus.cc:482] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:05.828262 18376 raft_consensus.cc:504] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.824491 18461 raft_consensus.cc:504] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.845297 18398 raft_consensus.cc:482] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.811903 18748 raft_consensus.cc:504] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.858750 18801 raft_consensus.cc:504] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.866986 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:05.868670 18801 leader_election.cc:288] T 0f7a6ca6100143da93759f88722f56e3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:05.846802 18827 raft_consensus.cc:504] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.848069 17943 raft_consensus.cc:504] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.862154 18792 raft_consensus.cc:3032] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 LEADER]: Stepping down as leader of term 9
I20260807 08:02:05.863008 18376 leader_election.cc:288] T d5c39633297442a8a7c63b17b17cbe54 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.863406 18461 leader_election.cc:288] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.863840 18398 raft_consensus.cc:504] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.868731 18748 leader_election.cc:288] T b1c0fb87eb274da9bfb29c3b9300dc1c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.884192 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:05.851567 18655 raft_consensus.cc:504] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.854506 18824 consensus_queue.cc:1046] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260807 08:02:05.886323 18824 raft_consensus.cc:3032] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 08:02:05.886706 18824 raft_consensus.cc:729] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:05.887135 18655 leader_election.cc:288] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.887611 18824 consensus_queue.cc:260] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.852196 18800 raft_consensus.cc:482] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.889335 18398 leader_election.cc:288] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:05.889029 18800 raft_consensus.cc:504] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.891577 18827 leader_election.cc:288] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:05.854666 18804 raft_consensus.cc:482] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.892334 18804 raft_consensus.cc:504] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.893756 17943 leader_election.cc:288] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.850569 18798 raft_consensus.cc:482] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.878268 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.894390 18792 raft_consensus.cc:729] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:05.895118 18798 raft_consensus.cc:504] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.896232 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.896025 18792 consensus_queue.cc:260] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.897193 18792 raft_consensus.cc:3037] T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:05.884070 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.914343 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:05.914891 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:05.878952 14618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42252 (request call id 667) took 2395 ms. Trace:
I20260807 08:02:05.914914 13709 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:05.916121 14618 rpcz_store.cc:276] 0807 08:02:03.483896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:03.736665 (+252769us) service_pool.cc:227] Handling call
0807 08:02:03.737049 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0807 08:02:03.841156 (+104107us) raft_consensus.cc:1520] Early marking committed up to index 7
0807 08:02:03.848754 (+ 7598us) log.cc:842] Serialized 14 byte log entry
0807 08:02:04.029734 (+180980us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0807 08:02:04.041732 (+ 11998us) log.cc:842] Serialized 26 byte log entry
0807 08:02:04.041873 (+ 141us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:02:04.041911 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:05.857315 (+1815404us) spinlock_profiling.cc:243] Waited 85.5 ms on lock 0x7b54005d1954. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:05.857365 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:05.857410 (+ 45us) raft_consensus.cc:1671] finished
0807 08:02:05.878696 (+ 21286us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:05.878882 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":13348,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":2699,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":12507,"spinlock_wait_cycles":196654976,"thread_start_us":76956,"threads_started":1,"wal-append.queue_time_us":235451,"wal-append.run_cpu_time_us":5011,"wal-append.run_wall_time_us":1442372}
I20260807 08:02:05.854275 18230 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.855369 18825 raft_consensus.cc:2781] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:02:05.921792 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:05.922439 14622 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:05.898646 13814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 762) took 1988 ms. Trace:
I20260807 08:02:05.850728 18760 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.901605 18196 raft_consensus.cc:3032] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:02:05.884546 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:05.902248 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:05.954165 18196 raft_consensus.cc:729] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:05.954555 14627 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
W20260807 08:02:05.954573 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 70376.7us
I20260807 08:02:05.955991 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:05.955339 18196 consensus_queue.cc:260] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.956772 14627 raft_consensus.cc:2454] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:02:05.884867 18773 consensus_queue.cc:1046] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260807 08:02:05.957985 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:05.958359 18773 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:02:05.958477 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e72e5415e504959b4763e580400f8ee" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:05.959050 18773 raft_consensus.cc:482] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:05.903038 18793 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:05.960054 13587 raft_consensus.cc:1226] T 35cf7298a676441c8d12e04b066ee036 P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:02:05.960381 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:05.961091 13590 raft_consensus.cc:2454] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 3.
I20260807 08:02:05.961810 13816 raft_consensus.cc:3037] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:05.962294 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:05.891335 18800 leader_election.cc:288] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:05.894618 18804 leader_election.cc:288] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:02:05.903987 18769 log.cc:924] Time spent T d44a80bdfaf145389eacef429f26eedc P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 1.638s user 0.014s sys 0.017s
I20260807 08:02:05.963197 13585 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:02:05.904358 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122c69d5e2045e380bdc9f6f8b487d2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:05.967140 18793 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.968183 13804 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:05.969359 13585 raft_consensus.cc:1261] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Refusing update from remote peer 22ff0a1f4a4241d794ab182432c85893: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260807 08:02:05.910458 18729 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.905221 13805 raft_consensus.cc:1226] T 2c1339f29eb34e05a3d97cef349b97c7 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:05.970227 18485 raft_consensus.cc:3032] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:02:05.970835 18485 raft_consensus.cc:729] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:05.971175 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6afc407b1b94edb94f8e7e8624c9221" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:05.971882 13805 raft_consensus.cc:2454] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 11.
I20260807 08:02:05.972736 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:02:05.909477 13540 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.140.66:44033 timed out after 1.973s (SENT) [suppressed 11 similar messages]
I20260807 08:02:05.904803 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:05.897680 18798 leader_election.cc:288] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:05.911448 13811 raft_consensus.cc:1261] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260807 08:02:05.919593 18836 raft_consensus.cc:482] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:05.993691 14583 leader_election.cc:302] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:05.993841 18836 raft_consensus.cc:504] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:05.995702 18793 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:05.995918 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.997648 18410 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:05.998342 18410 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.998921 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.999668 13703 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 5.
I20260807 08:02:06.000633 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.920621 18664 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:05.920948 18790 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:05.921201 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:05.922497 18784 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:05.939517 13769 leader_election.cc:302] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:05.941082 18445 raft_consensus.cc:482] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:06.006879 14629 raft_consensus.cc:2379] T e110e8648fd04684ae2bf4d0c71752a3 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 7: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:02:05.941109 18512 raft_consensus.cc:482] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260807 08:02:06.007336 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 67911.1us
I20260807 08:02:06.007234 18445 raft_consensus.cc:504] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.008327 18303 raft_consensus.cc:3032] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:02:05.942622 18741 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:06.008977 18303 raft_consensus.cc:729] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:05.946313 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:05.946465 18825 raft_consensus.cc:686] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [term 7 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:05.947453 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:05.949615 18767 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:06.020329 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.952004 17943 consensus_queue.cc:1046] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 08:02:05.953975 13661 leader_election.cc:398] T 466eca488bfd44d396c22e2396d1ef7e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Vote denied by peer 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123) with higher term. Message: Invalid argument: T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for earlier term 8. Current term is 9.
W20260807 08:02:06.029336 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 75569.3us
I20260807 08:02:05.960088 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:05.959519 18773 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:05.972023 18485 consensus_queue.cc:260] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:05.906675 13809 raft_consensus.cc:1226] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 9210dbc4cc9c45e5b26a463b705d362c for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:06.075306 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.076709 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:05.910828 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.975729 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.077843 13700 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:02:06.078018 13701 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:06.079205 13809 raft_consensus.cc:2379] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c in current term 8: Already voted for candidate 22ff0a1f4a4241d794ab182432c85893 in this term.
I20260807 08:02:06.079375 13701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:05.978135 13804 raft_consensus.cc:2454] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:02:06.081084 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbf945943e24c5dbc8b0a0f6c2fd2fd" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.082607 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:05.977967 13814 rpcz_store.cc:276] 0807 08:02:03.910227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:03.910373 (+ 146us) service_pool.cc:227] Handling call
0807 08:02:03.911219 (+ 846us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:05.583531 (+1672312us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:05.876266 (+292735us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400177854. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:05.898567 (+ 22301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3844272512}
I20260807 08:02:05.980489 13709 raft_consensus.cc:2454] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:02:05.981505 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:05.985343 18748 consensus_queue.cc:1046] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260807 08:02:06.084328 13807 raft_consensus.cc:3037] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Advancing to term 10
I20260807 08:02:05.970232 18729 raft_consensus.cc:504] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:05.990631 18753 log.cc:924] Time spent T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 1.979s user 0.005s sys 0.023s
I20260807 08:02:05.992332 18404 leader_election.cc:288] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.000787 14583 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:06.086781 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:05.997462 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.087498 13814 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:02:05.915069 18514 raft_consensus.cc:482] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:05.957243 18196 raft_consensus.cc:3037] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:06.004848 18410 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:06.007999 18512 raft_consensus.cc:504] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.009560 18793 raft_consensus.cc:2726] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260807 08:02:05.945649 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:05.945909 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab8c56441f84d86b5798c528463965c" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:05.941540 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:06.016731 13816 raft_consensus.cc:1261] T 069b769373e144929c9253bf3fdd0ab4 P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260807 08:02:05.997437 18836 leader_election.cc:288] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:06.009466 18741 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:06.020452 18767 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.028445 18445 leader_election.cc:288] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.036477 18303 consensus_queue.cc:260] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:06.039688 18478 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:06.030689 14622 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:02:06.047595 17943 raft_consensus.cc:3032] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:02:06.006654 18790 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.084172 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8727dbda122418487cb692370c2399e" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:05.979943 13811 pending_rounds.cc:76] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 4
I20260807 08:02:06.005959 18664 raft_consensus.cc:504] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:06.105628 14630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 790) took 1916 ms. Trace:
I20260807 08:02:06.134549 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8da02e8f51481cbcca504f8cff6f80" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.287103 13811 pending_rounds.cc:97] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260807 08:02:06.287496 13811 raft_consensus.cc:2866] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:02:06.021724 18423 raft_consensus.cc:2781] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 08:02:06.094691 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.287329 13812 raft_consensus.cc:2454] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:02:06.290367 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a55b0fbe6eb48649fd7cfc0be18010d" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.287592 13661 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:02:06.100082 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 99356.8us
I20260807 08:02:06.291325 13710 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:02:06.294752 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614fbd9fc524f61bb9d11ea324e783e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:06.295194 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:06.295830 13585 raft_consensus.cc:2454] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:02:06.201973 13817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59332 (request call id 768) took 2014 ms. Trace:
I20260807 08:02:06.296911 13817 rpcz_store.cc:276] 0807 08:02:04.187201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:04.190395 (+ 3194us) service_pool.cc:227] Handling call
0807 08:02:04.190686 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:05.853768 (+1663082us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:06.201634 (+347866us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540037c3d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:06.201890 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3818630016}
I20260807 08:02:06.158488 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e7d8fef17f4bdb92d0c21739339421" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.298851 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.288889 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.032011 18798 consensus_queue.cc:1046] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.353s
I20260807 08:02:06.290084 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1662) took 1137 ms. Trace:
I20260807 08:02:06.087386 18827 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:06.100984 13592 raft_consensus.cc:2454] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 4.
I20260807 08:02:06.298306 13702 raft_consensus.cc:2454] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:06.429949 13769 leader_election.cc:302] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893
I20260807 08:02:06.365020 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1339f29eb34e05a3d97cef349b97c7" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:06.428176 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:06.305125 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
W20260807 08:02:06.451694 13702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39150 (request call id 1894) took 1201 ms (client timeout 1555 ms). Trace:
I20260807 08:02:06.310523 13816 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 6.
I20260807 08:02:06.364144 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "9210dbc4cc9c45e5b26a463b705d362c" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.323566 14583 leader_election.cc:302] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
W20260807 08:02:06.377797 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 90291.5us
I20260807 08:02:06.356393 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.410866 14618 pending_rounds.cc:76] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Aborting all ops after (but not including) 8
I20260807 08:02:06.365399 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.429056 14629 raft_consensus.cc:1261] T d5c39633297442a8a7c63b17b17cbe54 P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260807 08:02:06.306071 13807 raft_consensus.cc:2454] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 10.
I20260807 08:02:06.224625 18770 raft_consensus.cc:482] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:06.334717 13809 raft_consensus.cc:1261] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Refusing update from remote peer 9210dbc4cc9c45e5b26a463b705d362c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260807 08:02:06.355831 13805 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9210dbc4cc9c45e5b26a463b705d362c in term 8.
I20260807 08:02:06.384582 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893"
I20260807 08:02:06.287113 14630 rpcz_store.cc:276] 0807 08:02:04.188930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:04.189046 (+ 116us) service_pool.cc:227] Handling call
0807 08:02:04.189353 (+ 307us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:02:04.895690 (+706337us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:04.895736 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:02:04.896508 (+ 772us) log.cc:842] Serialized 52 byte log entry
0807 08:02:04.897616 (+ 1108us) raft_consensus.cc:1631] Marking committed up to 6
0807 08:02:04.897667 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:06.105016 (+1207349us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b540020dad4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:06.105086 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:06.105147 (+ 61us) raft_consensus.cc:1671] finished
0807 08:02:06.105166 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:06.105553 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":28303,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"spinlock_wait_cycles":1470944640,"thread_start_us":66721,"threads_started":2,"wal-append.queue_time_us":1072,"wal-append.run_cpu_time_us":1715,"wal-append.run_wall_time_us":1037329}
I20260807 08:02:06.456877 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:06.455616 13812 raft_consensus.cc:1799] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Handling vote request from an unknown peer 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:06.454644 13814 raft_consensus.cc:2396] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260807 08:02:06.252029 18825 consensus_queue.cc:237] T d324e0b3eeee43549f0b430264c5d25b P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.468763 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:06.465982 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.179740 18370 raft_consensus.cc:3032] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Stepping down as leader of term 6
I20260807 08:02:06.455770 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.477056 13812 raft_consensus.cc:2396] T b09437796639470caac9548c71d72fb2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9210dbc4cc9c45e5b26a463b705d362c for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260807 08:02:06.476600 13809 pending_rounds.cc:76] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting all ops after (but not including) 4
I20260807 08:02:06.478567 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.490008 13809 pending_rounds.cc:97] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260807 08:02:06.489950 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.103374 18729 leader_election.cc:288] T 4c9ed355e84f4fc7a80a1d59a76a365c P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.465413 14618 pending_rounds.cc:97] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260807 08:02:06.418730 18793 raft_consensus.cc:482] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:06.408974 18515 raft_consensus.cc:482] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:06.466665 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bbc935aada4ceab2774e363186745d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.098953 18748 raft_consensus.cc:3032] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Stepping down as leader of term 4
W20260807 08:02:06.465101 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 141639us
W20260807 08:02:06.455114 13702 rpcz_store.cc:269] 0807 08:02:05.250264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:05.301605 (+ 51341us) service_pool.cc:227] Handling call
0807 08:02:06.158326 (+856721us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:02:06.451636 (+293310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5264768}
I20260807 08:02:06.350387 18482 leader_election.cc:288] T 6962d9b636574c5e80f4e57a227e52e2 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:06.500767 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e110e8648fd04684ae2bf4d0c71752a3" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:02:06.501201 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 71349.9us
I20260807 08:02:06.456283 13661 leader_election.cc:302] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
W20260807 08:02:06.385417 18831 log.cc:924] Time spent T 4614fbd9fc524f61bb9d11ea324e783e P 9210dbc4cc9c45e5b26a463b705d362c: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20260807 08:02:06.377480 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 468010us
I20260807 08:02:06.509176 13710 raft_consensus.cc:2396] T 76bbc935aada4ceab2774e363186745d P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260807 08:02:06.510666 13540 leader_election.cc:302] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:06.510972 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:06.490423 13809 raft_consensus.cc:2866] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260807 08:02:06.286904 17943 raft_consensus.cc:729] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:06.490831 13814 raft_consensus.cc:2379] T 2de183214b52498faef7f8a8f74a2adc P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 7: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:02:06.451262 13711 rpcz_store.cc:276] 0807 08:02:05.153073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:05.158987 (+ 5914us) service_pool.cc:227] Handling call
0807 08:02:05.159341 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:05.159704 (+ 363us) raft_consensus.cc:1520] Early marking committed up to index 9
0807 08:02:05.171975 (+ 12271us) log.cc:842] Serialized 14 byte log entry
0807 08:02:06.256136 (+1084161us) log.cc:842] Serialized 14 byte log entry
0807 08:02:06.289334 (+ 33198us) log.cc:842] Serialized 14 byte log entry
0807 08:02:06.289638 (+ 304us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:02:06.289697 (+ 59us) raft_consensus.cc:1631] Marking committed up to 9
0807 08:02:06.289768 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:06.289876 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:06.290015 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30113,"prepare.queue_time_us":12068,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":10481,"raft.queue_time_us":41142,"raft.run_cpu_time_us":488,"raft.run_wall_time_us":84651,"thread_start_us":1080296,"threads_started":3,"wal-append.queue_time_us":669446}
I20260807 08:02:06.322824 18410 consensus_queue.cc:1035] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.177s
I20260807 08:02:06.518743 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:06.519315 13711 raft_consensus.cc:3037] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:06.104214 18760 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.491820 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069b769373e144929c9253bf3fdd0ab4" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.500630 18748 raft_consensus.cc:729] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:06.492102 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.465858 14623 raft_consensus.cc:3037] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:06.100256 18584 raft_consensus.cc:2781] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:02:06.353854 18658 raft_consensus.cc:2726] T 5c683c4293d64060bd1a28772f77225a P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260807 08:02:06.528188 14620 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.023s sys 0.011s
I20260807 08:02:06.384852 18303 raft_consensus.cc:3037] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:06.360841 18664 leader_election.cc:288] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:06.355415 18790 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.359369 18512 leader_election.cc:288] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.357864 18767 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.385617 18378 raft_consensus.cc:482] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:06.353422 17585 leader_election.cc:288] T 0e72e5415e504959b4763e580400f8ee P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.454097 18782 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:06.486701 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.566740 13807 raft_consensus.cc:2454] T fb6a66ac7784477f904283ba3204bac6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
I20260807 08:02:06.567606 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c39633297442a8a7c63b17b17cbe54" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.568665 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.569175 13807 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:02:06.512599 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01b1964575c4708bf9760dff95db7f3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.569864 13809 raft_consensus.cc:2454] T a01b1964575c4708bf9760dff95db7f3 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 8.
I20260807 08:02:06.569932 14623 raft_consensus.cc:2454] T 00c1526fe2ae48f3a1bcce761ff91f72 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:06.513629 13811 raft_consensus.cc:2379] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77da774f2bb54cf28302d53d0db224ad in current term 6: Already voted for candidate 7a889001d6494fe5b7cbbe7da1dae302 in this term.
I20260807 08:02:06.514096 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74446812e814c7e9b4a1bd80bfb485d" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.351843 18741 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:06.516079 13702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966e99d2078c4b1c9fc921c1666b45a2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:06.571269 13812 raft_consensus.cc:2454] T b74446812e814c7e9b4a1bd80bfb485d P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:02:06.571581 13702 raft_consensus.cc:3037] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:06.516644 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
W20260807 08:02:06.516862 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 60663.4us
I20260807 08:02:06.289920 18423 raft_consensus.cc:482] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:06.521147 14630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:06.488883 18370 raft_consensus.cc:729] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9210dbc4cc9c45e5b26a463b705d362c, State: Running, Role: LEADER
I20260807 08:02:06.522554 18746 raft_consensus.cc:2726] T 069b769373e144929c9253bf3fdd0ab4 P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 08:02:06.494716 14618 raft_consensus.cc:2866] T 13ee9b6f7fa547a3badcb0eba0210119 P 22ff0a1f4a4241d794ab182432c85893 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260807 08:02:06.528591 14620 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s user 0.023s sys 0.011s
I20260807 08:02:06.533362 18760 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:06.528484 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09437796639470caac9548c71d72fb2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:06.531107 14629 raft_consensus.cc:1261] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260807 08:02:06.302989 18514 raft_consensus.cc:504] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:06.532114 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:06.574326 14630 raft_consensus.cc:2454] T c02eb1166e17491f82cfbd9c5cd030a2 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:06.577592 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.576421 14625 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:02:06.457790 18792 consensus_queue.cc:1035] T c8727dbda122418487cb692370c2399e P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260807 08:02:06.533954 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.584412 13814 raft_consensus.cc:2454] T 1588644cd4f943a7b5d776326af48218 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
I20260807 08:02:06.586858 13814 raft_consensus.cc:1261] T d6afc407b1b94edb94f8e7e8624c9221 P 7a889001d6494fe5b7cbbe7da1dae302 [term 11 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260807 08:02:06.535207 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.536170 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.537972 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e678279be64e6ca8a141d6b761e522" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:06.437196 18736 consensus_queue.cc:1035] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
W20260807 08:02:06.533133 18726 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.515s user 0.025s sys 0.027s
I20260807 08:02:06.519588 18801 raft_consensus.cc:2781] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:06.477308 18827 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.519309 18482 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:06.525054 18833 raft_consensus.cc:2726] T c78744cb93ef47e9aacb62cbeb454457 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260807 08:02:06.528239 18584 raft_consensus.cc:482] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:06.509510 18515 raft_consensus.cc:504] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.579825 13807 raft_consensus.cc:1261] T 04e678279be64e6ca8a141d6b761e522 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Refusing update from remote peer 77da774f2bb54cf28302d53d0db224ad: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260807 08:02:06.573963 18658 raft_consensus.cc:482] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:06.642458 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f6b5b24a8345d0ba918651b4a2c85f" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.500032 18793 raft_consensus.cc:504] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.524371 18794 consensus_queue.cc:1035] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.044s
I20260807 08:02:06.527803 18729 consensus_queue.cc:1046] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260807 08:02:06.577133 18423 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:02:06.493048 18770 raft_consensus.cc:504] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
W20260807 08:02:06.617730 18726 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.601s user 0.028s sys 0.027s
I20260807 08:02:06.615450 18746 raft_consensus.cc:482] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:06.616539 18801 raft_consensus.cc:482] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22ff0a1f4a4241d794ab182432c85893)
I20260807 08:02:06.640570 18584 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:06.564627 18378 raft_consensus.cc:504] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.640964 18482 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.576841 18782 raft_consensus.cc:504] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.596570 18760 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.664021 18801 raft_consensus.cc:3037] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:06.658675 18793 leader_election.cc:288] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.579288 17943 consensus_queue.cc:260] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.648351 18658 raft_consensus.cc:504] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:06.615039 18748 consensus_queue.cc:260] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.672125 18726 raft_consensus.cc:3032] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Stepping down as leader of term 7
I20260807 08:02:06.667273 18746 raft_consensus.cc:504] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:06.695331 18726 raft_consensus.cc:729] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:06.757542 13701 raft_consensus.cc:2454] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 7.
W20260807 08:02:06.758172 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 252610us
I20260807 08:02:06.562604 13711 raft_consensus.cc:2454] T 9b4b1bd6a74d452e860b1420b0ef1434 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:02:06.579314 13661 leader_election.cc:302] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:06.765285 13769 leader_election.cc:302] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:06.766341 13711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37920c01cb94122bd1c5a0bb331be31" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:06.768524 13808 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:02:06.767715 13711 raft_consensus.cc:3037] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 5 FOLLOWER]: Advancing to term 6
I20260807 08:02:06.776329 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.766155 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.779313 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de183214b52498faef7f8a8f74a2adc" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.588539 13703 raft_consensus.cc:2454] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 6.
I20260807 08:02:06.781441 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ccb94230614cc48e846580ad82e994" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.783175 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2acf2131e15e4fc9b2be438b9f3579b1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.635046 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ee9b6f7fa547a3badcb0eba0210119" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.636299 13811 raft_consensus.cc:1226] T b122c69d5e2045e380bdc9f6f8b487d2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260807 08:02:06.591131 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70356515c14242539aa745f7e028f6b6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
W20260807 08:02:06.842012 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 206880us
I20260807 08:02:06.842727 13710 raft_consensus.cc:2454] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:06.843766 13661 leader_election.cc:302] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:06.844421 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17c7c715cf146728d57366f8e070d31" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.845816 13769 leader_election.cc:302] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:06.846161 18793 consensus_queue.cc:1035] T d5c39633297442a8a7c63b17b17cbe54 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260807 08:02:06.847616 13769 leader_election.cc:302] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302; no voters:
I20260807 08:02:06.849311 18836 raft_consensus.cc:2781] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election won for term 7
I20260807 08:02:06.781281 13808 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 5.
I20260807 08:02:06.850139 18836 raft_consensus.cc:686] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 LEADER]: Becoming Leader. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:06.851277 18792 raft_consensus.cc:2781] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:06.852034 18792 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:06.852571 18792 raft_consensus.cc:3037] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:06.812417 18782 leader_election.cc:288] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:06.810586 13703 raft_consensus.cc:2454] T 2acf2131e15e4fc9b2be438b9f3579b1 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 8.
I20260807 08:02:06.786845 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c1526fe2ae48f3a1bcce761ff91f72" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.854816 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e3dc2093eb4153a975824017d2aafb" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.855192 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.854933 18726 consensus_queue.cc:260] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:06.856561 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02eb1166e17491f82cfbd9c5cd030a2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.856704 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d324e0b3eeee43549f0b430264c5d25b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:06.828756 18857 raft_consensus.cc:2781] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:06.857889 18857 raft_consensus.cc:482] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:06.831744 18770 leader_election.cc:288] T 466eca488bfd44d396c22e2396d1ef7e P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:06.832469 18748 raft_consensus.cc:3037] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:02:06.841333 18427 raft_consensus.cc:2726] T cc32413495ee41909c7c5494c8095084 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:06.850222 18512 raft_consensus.cc:2781] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:06.865626 18512 raft_consensus.cc:482] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9210dbc4cc9c45e5b26a463b705d362c)
I20260807 08:02:06.866142 18512 raft_consensus.cc:3037] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:06.867578 18378 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:06.826217 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6962d9b636574c5e80f4e57a227e52e2" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.869572 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.871012 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.828238 18515 leader_election.cc:288] T 06a735af95cd49e1942d669d2e5d91c6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.867411 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc32413495ee41909c7c5494c8095084" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:06.875928 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa406406335b4c9688263b810ce070a9" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c"
I20260807 08:02:06.876950 13703 raft_consensus.cc:2379] T fa406406335b4c9688263b810ce070a9 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a889001d6494fe5b7cbbe7da1dae302 in current term 6: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:06.659106 18370 consensus_queue.cc:260] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:06.878435 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5ccbaa50ae4451a35038284693254c" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.879215 18370 raft_consensus.cc:3037] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:06.880208 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6a66ac7784477f904283ba3204bac6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.880872 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa5c12b1b5e4feea56543c706f21ec2" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.881724 13708 raft_consensus.cc:2454] T caa5c12b1b5e4feea56543c706f21ec2 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:06.852288 18836 consensus_queue.cc:237] T 00c1526fe2ae48f3a1bcce761ff91f72 P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.859537 13710 raft_consensus.cc:2454] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 10.
I20260807 08:02:06.850186 18767 consensus_queue.cc:1035] T 4c9ed355e84f4fc7a80a1d59a76a365c P 7a889001d6494fe5b7cbbe7da1dae302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260807 08:02:06.899507 18658 leader_election.cc:288] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:06.900656 18746 leader_election.cc:288] T 2c1339f29eb34e05a3d97cef349b97c7 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:06.904554 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7a6ca6100143da93759f88722f56e3" candidate_uuid: "77da774f2bb54cf28302d53d0db224ad" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.906203 18514 leader_election.cc:288] T 23f6b5b24a8345d0ba918651b4a2c85f P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:06.911286 18482 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:02:06.912601 18848 log.cc:924] Time spent T 04e678279be64e6ca8a141d6b761e522 P 22ff0a1f4a4241d794ab182432c85893: Append to log took a long time: real 0.234s user 0.000s sys 0.001s
I20260807 08:02:06.914901 18760 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:06.916754 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.922092 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccaeb8c899348e7959597174d89ed01" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.923446 13710 raft_consensus.cc:2396] T 1ccaeb8c899348e7959597174d89ed01 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260807 08:02:06.925168 13710 raft_consensus.cc:1226] T d2dfc89fb081405f96a0b53d76aeea38 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 LEADER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:06.925725 18800 raft_consensus.cc:3032] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 08:02:06.858544 18857 raft_consensus.cc:3037] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:06.936283 18196 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:06.941609 13809 raft_consensus.cc:3037] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
W20260807 08:02:06.951767 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 57694.8us
I20260807 08:02:06.954659 13769 leader_election.cc:302] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:06.956274 13769 leader_election.cc:302] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
W20260807 08:02:06.958997 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 115531us
I20260807 08:02:06.933926 18800 raft_consensus.cc:729] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7a889001d6494fe5b7cbbe7da1dae302, State: Running, Role: LEADER
I20260807 08:02:06.963330 18600 raft_consensus.cc:2781] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:06.964007 18600 raft_consensus.cc:482] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:06.964570 18600 raft_consensus.cc:3037] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:06.964293 18800 consensus_queue.cc:260] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } attrs { replace: true } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:06.958503 13703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1588644cd4f943a7b5d776326af48218" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "9210dbc4cc9c45e5b26a463b705d362c" is_pre_election: true
I20260807 08:02:06.974051 13711 raft_consensus.cc:2454] T d37920c01cb94122bd1c5a0bb331be31 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 6.
I20260807 08:02:06.965797 13705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1676) took 1062 ms. Trace:
I20260807 08:02:06.974934 13711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45382 (request call id 518) took 1055 ms. Trace:
I20260807 08:02:07.007061 13809 raft_consensus.cc:2454] T d324e0b3eeee43549f0b430264c5d25b P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:02:06.979182 13711 rpcz_store.cc:276] 0807 08:02:05.919887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:06.760740 (+840853us) service_pool.cc:227] Handling call
0807 08:02:06.974854 (+214114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:02:06.976228 13705 rpcz_store.cc:276] 0807 08:02:05.873199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:05.907809 (+ 34610us) service_pool.cc:227] Handling call
0807 08:02:06.904093 (+996284us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b5800003f98. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b19e87f11 00007f6b1a00b851 00007f6b1a00b702 00007f6b22efbb09 00007f6b21e37730 00007f6b21efd79b 00007f6b21efd821 00007f6b21ead1b0 00007f6b21eaca94 00007f6b1b594c04 00007f6b1b594b92
0807 08:02:06.935727 (+ 31634us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7408384}
I20260807 08:02:07.013137 13709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 519) took 1009 ms. Trace:
I20260807 08:02:07.013748 13709 rpcz_store.cc:276] 0807 08:02:05.920258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:06.900429 (+980171us) service_pool.cc:227] Handling call
0807 08:02:06.900926 (+ 497us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:06.901358 (+ 432us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:06.901399 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:02:06.901455 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:02:06.901508 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:06.929317 (+ 27809us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:06.929464 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:02:07.020448 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9ed355e84f4fc7a80a1d59a76a365c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:07.019740 13812 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 LEADER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
W20260807 08:02:07.026803 18773 consensus_meta.cc:287] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.978s user 0.018s sys 0.002s
I20260807 08:02:07.027407 18773 raft_consensus.cc:504] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.039222 18726 raft_consensus.cc:3037] T b122c69d5e2045e380bdc9f6f8b487d2 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Advancing to term 8
I20260807 08:02:07.022670 18800 raft_consensus.cc:3037] T 0248ade936924e628c46b8de6252af1b P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:07.052253 18521 raft_consensus.cc:2781] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 08:02:07.061010 18521 raft_consensus.cc:482] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:07.061579 18521 raft_consensus.cc:3037] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Advancing to term 9
W20260807 08:02:07.062812 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 502917us
I20260807 08:02:07.064451 13540 leader_election.cc:302] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:07.065474 18482 consensus_queue.cc:1035] T 4fbf945943e24c5dbc8b0a0f6c2fd2fd P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 08:02:07.066635 18801 raft_consensus.cc:504] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.067987 18482 raft_consensus.cc:2726] T e67bd63471c94c19a092c8f1aef64408 P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260807 08:02:07.071430 18478 consensus_meta.cc:287] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.749s user 0.013s sys 0.000s
I20260807 08:02:07.071285 18512 raft_consensus.cc:504] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.072093 13708 raft_consensus.cc:2454] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 6.
I20260807 08:02:07.074285 18827 leader_election.cc:288] T 0248ade936924e628c46b8de6252af1b P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:02:07.074561 18303 consensus_meta.cc:287] T 1a55b0fbe6eb48649fd7cfc0be18010d P 77da774f2bb54cf28302d53d0db224ad: Time spent flushing consensus metadata: real 0.533s user 0.008s sys 0.008s
I20260807 08:02:07.075254 17585 raft_consensus.cc:482] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:07.076018 17585 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.080494 17585 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:07.084790 13703 raft_consensus.cc:1226] T 1a55b0fbe6eb48649fd7cfc0be18010d P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 7. Current term is 8. Ops: []
I20260807 08:02:07.085613 13703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39134 (request call id 1688) took 1087 ms. Trace:
I20260807 08:02:07.085935 13703 rpcz_store.cc:276] 0807 08:02:05.998378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:07.003379 (+1005001us) service_pool.cc:227] Handling call
0807 08:02:07.084527 (+ 81148us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:07.085291 (+ 764us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:07.085533 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:02:07.088435 13710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45382 (request call id 526) took 1115 ms. Trace:
I20260807 08:02:07.088881 13710 rpcz_store.cc:276] 0807 08:02:05.943930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:06.957538 (+1013608us) service_pool.cc:227] Handling call
0807 08:02:07.003240 (+ 45702us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:07.053465 (+ 50225us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:07.053507 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0807 08:02:07.059043 (+ 5536us) raft_consensus.cc:1631] Marking committed up to 7
0807 08:02:07.059102 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:07.059182 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:07.059356 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:02:07.093439 13711 raft_consensus.cc:1261] T b17c7c715cf146728d57366f8e070d31 P 9210dbc4cc9c45e5b26a463b705d362c [term 10 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260807 08:02:07.095894 18512 leader_election.cc:288] T c02eb1166e17491f82cfbd9c5cd030a2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:07.125068 13805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54086 (request call id 2170) took 1125 ms. Trace:
I20260807 08:02:07.125572 13805 rpcz_store.cc:276] 0807 08:02:05.969227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:05.972492 (+ 3265us) service_pool.cc:227] Handling call
0807 08:02:07.091236 (+1118744us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b54003e3d54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:07.094469 (+ 3233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":879490944}
I20260807 08:02:07.126976 13817 raft_consensus.cc:1226] T 35cf7298a676441c8d12e04b066ee036 P 7a889001d6494fe5b7cbbe7da1dae302 [term 5 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 4. Current term is 5. Ops: []
I20260807 08:02:07.128402 13701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1689) took 1129 ms. Trace:
I20260807 08:02:07.128762 13705 raft_consensus.cc:1261] T 6962d9b636574c5e80f4e57a227e52e2 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Refusing update from remote peer 7a889001d6494fe5b7cbbe7da1dae302: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260807 08:02:07.128924 13701 rpcz_store.cc:276] 0807 08:02:05.998809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:06.079052 (+ 80243us) service_pool.cc:227] Handling call
0807 08:02:07.128103 (+1049051us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b5400292554. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:07.128309 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1556293376}
I20260807 08:02:07.130664 14620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42236 (request call id 653) took 1230 ms. Trace:
W20260807 08:02:07.137230 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 121121us
I20260807 08:02:07.131704 14619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42236 (request call id 656) took 1230 ms. Trace:
W20260807 08:02:07.153981 13769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 61670.3us
I20260807 08:02:07.145805 14619 rpcz_store.cc:276] 0807 08:02:05.901138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:05.904593 (+ 3455us) service_pool.cc:227] Handling call
0807 08:02:07.131386 (+1226793us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b54004515d4. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6f2ae 00007f6b21eacc27 00007f6b1b594c04 00007f6b1b594b92 00007f6b1b594ac1 00007f6b1b594a3c 00007f6b1b593cf2 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:07.131522 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2196091776}
W20260807 08:02:07.165369 13540 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.66:44033, user_credentials={real_user=slave}} blocked reactor thread for 101099us
I20260807 08:02:07.131139 14620 rpcz_store.cc:276] 0807 08:02:05.900141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:05.902834 (+ 2693us) service_pool.cc:227] Handling call
0807 08:02:05.903158 (+ 324us) raft_consensus.cc:1457] Updating replica for 2 ops
0807 08:02:05.903544 (+ 386us) raft_consensus.cc:1520] Early marking committed up to index 0
0807 08:02:05.903574 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0807 08:02:05.904109 (+ 535us) log.cc:842] Serialized 52 byte log entry
0807 08:02:06.574796 (+670687us) raft_consensus.cc:1631] Marking committed up to 8
0807 08:02:07.091494 (+516698us) log.cc:842] Serialized 14 byte log entry
0807 08:02:07.130038 (+ 38544us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:07.130151 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0807 08:02:07.130197 (+ 46us) raft_consensus.cc:1671] finished
0807 08:02:07.130216 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0807 08:02:07.130381 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3748,"prepare.queue_time_us":557011,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":6174,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":11744,"thread_start_us":1110574,"threads_started":2,"wal-append.queue_time_us":749956}
I20260807 08:02:07.136318 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232127da9f574f41a5a8087ae775ef6b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:07.134326 18485 raft_consensus.cc:3037] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:07.160743 18773 leader_election.cc:288] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:07.177282 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:07.177888 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fcdcbbf735436893ce5c13fd2eb1f1" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:07.178838 18801 leader_election.cc:288] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
W20260807 08:02:07.193557 13702 consensus_meta.cc:287] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c: Time spent flushing consensus metadata: real 0.516s user 0.004s sys 0.012s
I20260807 08:02:07.203424 13817 raft_consensus.cc:1226] T 9ab8c56441f84d86b5798c528463965c P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Rejecting Update request from peer 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7. Ops: []
I20260807 08:02:07.180090 18792 raft_consensus.cc:504] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.204746 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4b1bd6a74d452e860b1420b0ef1434" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302"
I20260807 08:02:07.204945 18864 consensus_queue.cc:1035] T caa5c12b1b5e4feea56543c706f21ec2 P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260807 08:02:07.205595 13812 raft_consensus.cc:2362] T 9b4b1bd6a74d452e860b1420b0ef1434 P 7a889001d6494fe5b7cbbe7da1dae302 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8. Re-sending same reply.
I20260807 08:02:07.206296 13661 leader_election.cc:302] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 77da774f2bb54cf28302d53d0db224ad
I20260807 08:02:07.207983 13661 leader_election.cc:302] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9210dbc4cc9c45e5b26a463b705d362c; no voters: 22ff0a1f4a4241d794ab182432c85893, 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:07.207861 18857 raft_consensus.cc:504] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.209712 18521 raft_consensus.cc:504] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:07.210937 17585 raft_consensus.cc:2726] T 38e7d8fef17f4bdb92d0c21739339421 P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260807 08:02:07.210659 18268 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.906s user 0.023s sys 0.058s
I20260807 08:02:07.227370 17943 raft_consensus.cc:3037] T e110e8648fd04684ae2bf4d0c71752a3 P 9210dbc4cc9c45e5b26a463b705d362c [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:07.229867 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466eca488bfd44d396c22e2396d1ef7e" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:07.233965 13804 raft_consensus.cc:1226] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 9 FOLLOWER]: Rejecting Update request from peer 77da774f2bb54cf28302d53d0db224ad for earlier term 8. Current term is 9. Ops: []
W20260807 08:02:07.234206 18584 consensus_meta.cc:287] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.562s user 0.005s sys 0.011s
I20260807 08:02:07.235857 18584 raft_consensus.cc:504] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.236552 18857 leader_election.cc:288] T 5c683c4293d64060bd1a28772f77225a P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:07.225962 13585 raft_consensus.cc:2454] T 232127da9f574f41a5a8087ae775ef6b P 77da774f2bb54cf28302d53d0db224ad [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 7.
I20260807 08:02:07.239867 13807 raft_consensus.cc:2354] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 for earlier term 6. Current term is 7.
I20260807 08:02:07.212182 13702 raft_consensus.cc:2454] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da774f2bb54cf28302d53d0db224ad in term 9.
W20260807 08:02:07.238739 18268 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.984s user 0.023s sys 0.065s
I20260807 08:02:07.244946 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248ade936924e628c46b8de6252af1b" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:07.245266 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e37d72ff9b640289fad70dde36572fe" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:07.246357 13702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39134 (request call id 1683) took 1339 ms. Trace:
I20260807 08:02:07.247020 13769 leader_election.cc:302] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:07.249294 13540 leader_election.cc:302] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 7a889001d6494fe5b7cbbe7da1dae302, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:07.239689 18773 raft_consensus.cc:482] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:07.269119 13702 rpcz_store.cc:276] 0807 08:02:05.906998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:06.515839 (+608841us) service_pool.cc:227] Handling call
0807 08:02:07.246243 (+730404us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 08:02:07.253549 14583 leader_election.cc:302] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:07.255089 18709 raft_consensus.cc:2741] T 966e99d2078c4b1c9fc921c1666b45a2 P 9210dbc4cc9c45e5b26a463b705d362c [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260807 08:02:07.256840 13804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54072 (request call id 1734) took 1263 ms. Trace:
W20260807 08:02:07.257619 18423 consensus_meta.cc:287] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893: Time spent flushing consensus metadata: real 0.593s user 0.010s sys 0.004s
I20260807 08:02:07.259048 18824 raft_consensus.cc:3037] T 06a735af95cd49e1942d669d2e5d91c6 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Advancing to term 6
W20260807 08:02:07.261241 18890 log.cc:924] Time spent T 1ccaeb8c899348e7959597174d89ed01 P 7a889001d6494fe5b7cbbe7da1dae302: Append to log took a long time: real 0.107s user 0.000s sys 0.002s
I20260807 08:02:07.262620 18600 raft_consensus.cc:504] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:07.265736 18792 leader_election.cc:288] T 25fcdcbbf735436893ce5c13fd2eb1f1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033), 22ff0a1f4a4241d794ab182432c85893 (127.11.140.68:40123)
I20260807 08:02:07.253232 18196 raft_consensus.cc:2726] T e4e01d60f0064bb3958a65250e430990 P 9210dbc4cc9c45e5b26a463b705d362c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260807 08:02:07.295310 18600 leader_election.cc:288] T 70356515c14242539aa745f7e028f6b6 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 7 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:07.271165 13590 raft_consensus.cc:2379] T 0248ade936924e628c46b8de6252af1b P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 9: Already voted for candidate 77da774f2bb54cf28302d53d0db224ad in this term.
I20260807 08:02:07.271679 13804 rpcz_store.cc:276] 0807 08:02:05.993008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:06.384781 (+391773us) service_pool.cc:227] Handling call
0807 08:02:07.122171 (+737390us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:07.234466 (+112295us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:07.256061 (+ 21595us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b54003e3d54. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:07.256619 (+ 558us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b54003e3d50. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6aa09 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86 00007f6b1b39472f
0807 08:02:07.256776 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1796799744}
I20260807 08:02:07.275220 13579 raft_consensus.cc:3037] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:07.298311 18521 leader_election.cc:288] T 2acf2131e15e4fc9b2be438b9f3579b1 P 7a889001d6494fe5b7cbbe7da1dae302 [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:07.275086 18773 raft_consensus.cc:504] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.275103 18642 raft_consensus.cc:2781] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 08:02:07.276901 18729 raft_consensus.cc:2726] T 88e3dc2093eb4153a975824017d2aafb P 77da774f2bb54cf28302d53d0db224ad [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 08:02:07.277544 13540 leader_election.cc:302] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters: 7a889001d6494fe5b7cbbe7da1dae302
I20260807 08:02:07.278980 18423 raft_consensus.cc:504] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.302950 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c0fb87eb274da9bfb29c3b9300dc1c" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a889001d6494fe5b7cbbe7da1dae302" is_pre_election: true
I20260807 08:02:07.303615 14626 raft_consensus.cc:1226] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Rejecting Update request from peer 7a889001d6494fe5b7cbbe7da1dae302 for earlier term 3. Current term is 4. Ops: []
I20260807 08:02:07.303793 13816 raft_consensus.cc:2379] T b1c0fb87eb274da9bfb29c3b9300dc1c P 7a889001d6494fe5b7cbbe7da1dae302 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22ff0a1f4a4241d794ab182432c85893 in current term 10: Already voted for candidate 9210dbc4cc9c45e5b26a463b705d362c in this term.
I20260807 08:02:07.304327 18423 leader_election.cc:288] T e5ef275b7c0a45619cbc05266aea24dc P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 4 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67:40969)
I20260807 08:02:07.305353 14626 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42248 (request call id 794) took 1010 ms. Trace:
I20260807 08:02:07.305721 14626 rpcz_store.cc:276] 0807 08:02:06.294688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 08:02:06.294849 (+ 161us) service_pool.cc:227] Handling call
0807 08:02:06.295135 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0807 08:02:07.304051 (+1008916us) raft_consensus.cc:1683] Filling consensus response to leader.
0807 08:02:07.305086 (+ 1035us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400696254. stack: 00007f6b1a011033 00007f6b1a010f30 00007f6b1996f534 00007f6b2367b8b4 00007f6b2367b849 00007f6b1cef0471 00007f6b1cf6bf51 00007f6b1cf6a99e 00007f6b21eac2e9 00007f6b1b592684 00007f6b1b592612 00007f6b1b592541 00007f6b1b5924bc 00007f6b1b591772 00007f6b1b39594c 00007f6b1b394d86
0807 08:02:07.305297 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2318330240}
I20260807 08:02:07.307897 18864 raft_consensus.cc:2781] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 10 FOLLOWER]: Leader election won for term 10
I20260807 08:02:07.308490 18864 raft_consensus.cc:686] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [term 10 LEADER]: Becoming Leader. State: Replica: 77da774f2bb54cf28302d53d0db224ad, State: Running, Role: LEADER
I20260807 08:02:07.308849 18865 consensus_queue.cc:1035] T 466eca488bfd44d396c22e2396d1ef7e P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260807 08:02:07.309417 18864 consensus_queue.cc:237] T d44a80bdfaf145389eacef429f26eedc P 77da774f2bb54cf28302d53d0db224ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } }
I20260807 08:02:07.300447 18642 raft_consensus.cc:482] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77da774f2bb54cf28302d53d0db224ad)
I20260807 08:02:07.312626 18642 raft_consensus.cc:3037] T caa5c12b1b5e4feea56543c706f21ec2 P 7a889001d6494fe5b7cbbe7da1dae302 [term 6 FOLLOWER]: Advancing to term 7
I20260807 08:02:07.312736 13579 raft_consensus.cc:2454] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 9.
I20260807 08:02:07.279927 18584 leader_election.cc:288] T e4e01d60f0064bb3958a65250e430990 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 election: Requested vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:07.280364 13700 raft_consensus.cc:2454] T 2de183214b52498faef7f8a8f74a2adc P 9210dbc4cc9c45e5b26a463b705d362c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 7.
I20260807 08:02:07.281804 13432 catalog_manager.cc:5495] T d6afc407b1b94edb94f8e7e8624c9221 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 7 to 9, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 9 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
W20260807 08:02:07.322674 13661 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.68:40123, user_credentials={real_user=slave}} blocked reactor thread for 113213us
W20260807 08:02:07.276479 14580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 268884us
I20260807 08:02:07.285509 14583 leader_election.cc:302] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:07.302651 18773 leader_election.cc:288] T c78744cb93ef47e9aacb62cbeb454457 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:07.285147 18748 raft_consensus.cc:2781] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 08:02:07.322038 13432 catalog_manager.cc:5495] T b09437796639470caac9548c71d72fb2 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 3 to 4, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 4 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260807 08:02:07.324646 18748 raft_consensus.cc:482] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:02:07.321125 13540 leader_election.cc:302] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad; no voters: 22ff0a1f4a4241d794ab182432c85893, 9210dbc4cc9c45e5b26a463b705d362c
I20260807 08:02:07.325062 18748 raft_consensus.cc:3037] T 4c9ed355e84f4fc7a80a1d59a76a365c P 22ff0a1f4a4241d794ab182432c85893 [term 4 FOLLOWER]: Advancing to term 5
I20260807 08:02:07.328413 13432 catalog_manager.cc:5495] T 88e3dc2093eb4153a975824017d2aafb P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260807 08:02:07.330423 13432 catalog_manager.cc:5495] T 70356515c14242539aa745f7e028f6b6 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 4 to 5, leader changed from 77da774f2bb54cf28302d53d0db224ad (127.11.140.65) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 5 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67" port: 40969 } health_report { overall_health: UNKNOWN } } }
W20260807 08:02:07.332269 14583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.65:41037, user_credentials={real_user=slave}} blocked reactor thread for 46892.1us
I20260807 08:02:07.332391 18773 raft_consensus.cc:2781] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 08:02:07.332865 18773 raft_consensus.cc:482] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7a889001d6494fe5b7cbbe7da1dae302)
I20260807 08:02:07.333238 18773 raft_consensus.cc:3037] T 232127da9f574f41a5a8087ae775ef6b P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Advancing to term 8
W20260807 08:02:07.333626 13657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.140.67:40969, user_credentials={real_user=slave}} blocked reactor thread for 119857us
I20260807 08:02:07.334120 14583 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ff0a1f4a4241d794ab182432c85893, 77da774f2bb54cf28302d53d0db224ad; no voters:
I20260807 08:02:07.336236 18423 raft_consensus.cc:2781] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 9 FOLLOWER]: Leader election won for term 9
I20260807 08:02:07.336431 11825 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260807 08:02:07.336778 18423 raft_consensus.cc:686] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [term 9 LEADER]: Becoming Leader. State: Replica: 22ff0a1f4a4241d794ab182432c85893, State: Running, Role: LEADER
I20260807 08:02:07.337893 18423 consensus_queue.cc:237] T 2e37d72ff9b640289fad70dde36572fe P 22ff0a1f4a4241d794ab182432c85893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.342707 18729 raft_consensus.cc:2726] T 46d5ddf3f7d84e0ebda02d99abff4f0e P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260807 08:02:07.346580 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e01d60f0064bb3958a65250e430990" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:07.347429 13590 raft_consensus.cc:3037] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Advancing to term 9
I20260807 08:02:07.348731 13540 leader_election.cc:302] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:07.349484 18729 raft_consensus.cc:2741] T 2e37d72ff9b640289fad70dde36572fe P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260807 08:02:07.353060 13540 leader_election.cc:302] T fa406406335b4c9688263b810ce070a9 P 77da774f2bb54cf28302d53d0db224ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da774f2bb54cf28302d53d0db224ad, 9210dbc4cc9c45e5b26a463b705d362c; no voters:
I20260807 08:02:07.356205 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c683c4293d64060bd1a28772f77225a" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:07.360800 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ef275b7c0a45619cbc05266aea24dc" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad"
I20260807 08:02:07.360967 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78744cb93ef47e9aacb62cbeb454457" candidate_uuid: "22ff0a1f4a4241d794ab182432c85893" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77da774f2bb54cf28302d53d0db224ad" is_pre_election: true
I20260807 08:02:07.361441 13580 raft_consensus.cc:3037] T e5ef275b7c0a45619cbc05266aea24dc P 77da774f2bb54cf28302d53d0db224ad [term 3 FOLLOWER]: Advancing to term 4
I20260807 08:02:07.361992 13592 raft_consensus.cc:2454] T c78744cb93ef47e9aacb62cbeb454457 P 77da774f2bb54cf28302d53d0db224ad [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 8.
I20260807 08:02:07.363909 18891 raft_consensus.cc:482] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:02:07.364457 18891 raft_consensus.cc:504] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } } peers { permanent_uuid: "22ff0a1f4a4241d794ab182432c85893" member_type: VOTER last_known_addr { host: "127.11.140.68" port: 40123 } attrs { promote: false } }
I20260807 08:02:07.366310 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cf7298a676441c8d12e04b066ee036" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:07.366653 18891 leader_election.cc:288] T d2dfc89fb081405f96a0b53d76aeea38 P 22ff0a1f4a4241d794ab182432c85893 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77da774f2bb54cf28302d53d0db224ad (127.11.140.65:41037), 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66:44033)
I20260807 08:02:07.366958 14626 raft_consensus.cc:2454] T 35cf7298a676441c8d12e04b066ee036 P 22ff0a1f4a4241d794ab182432c85893 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a889001d6494fe5b7cbbe7da1dae302 in term 5.
I20260807 08:02:07.371914 14626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a735af95cd49e1942d669d2e5d91c6" candidate_uuid: "7a889001d6494fe5b7cbbe7da1dae302" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22ff0a1f4a4241d794ab182432c85893" is_pre_election: true
I20260807 08:02:07.377270 13590 raft_consensus.cc:2454] T e4e01d60f0064bb3958a65250e430990 P 77da774f2bb54cf28302d53d0db224ad [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ff0a1f4a4241d794ab182432c85893 in term 9.
I20260807 08:02:07.372752 13432 catalog_manager.cc:5495] T b122c69d5e2045e380bdc9f6f8b487d2 P 9210dbc4cc9c45e5b26a463b705d362c reported cstate change: term changed from 3 to 6, leader changed from 7a889001d6494fe5b7cbbe7da1dae302 (127.11.140.67) to 9210dbc4cc9c45e5b26a463b705d362c (127.11.140.66). New cstate: current_term: 6 leader_uuid: "9210dbc4cc9c45e5b26a463b705d362c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da774f2bb54cf28302d53d0db224ad" member_type: VOTER last_known_addr { host: "127.11.140.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9210dbc4cc9c45e5b26a463b705d362c" member_type: VOTER last_known_addr { host: "127.11.140.66" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a889001d6494fe5b7cbbe7da1dae302" member_type: VOTER last_known_addr { host: "127.11.140.67"